2024-03-25T18:07:00.861 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T18:07:00.862 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621321 --owner scheduled_amathuri@teuthology --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621321/orig.config.yaml 2024-03-25T18:07:00.903 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:07:00.982 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621321 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621321' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi044.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJiZboMgAKUllmKZXbQtFQg4Vo+6OojgRvyCN6U7ovHyVJwE05w3X98t/cBtzDQ24ni5kRFzxbZOCHzqX/RoBeY= smithi111.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB6T5MuIscewO93LAeMlMw2BXb8rpyN4fq4Q2BO2Ncz/pFnpvRkgRfFEX/tARk77Vj9MbOnGxQm1YbS2Y7/PybQ= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T18:07:00.982 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T18:07:00.982 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T18:07:00.983 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T18:07:00.984 INFO:teuthology.task.internal:Checking packages... 2024-03-25T18:07:01.006 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T18:07:01.006 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T18:07:01.006 INFO:teuthology.packaging:ref: None 2024-03-25T18:07:01.006 INFO:teuthology.packaging:tag: None 2024-03-25T18:07:01.006 INFO:teuthology.packaging:branch: main 2024-03-25T18:07:01.006 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:07:01.006 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T18:07:01.343 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T18:07:01.345 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T18:07:01.351 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T18:07:01.351 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T18:07:01.372 INFO:teuthology.task.internal:Saving configuration 2024-03-25T18:07:01.387 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T18:07:01.393 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:07:01.418 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi044.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621321', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:01:38.261477', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJiZboMgAKUllmKZXbQtFQg4Vo+6OojgRvyCN6U7ovHyVJwE05w3X98t/cBtzDQ24ni5kRFzxbZOCHzqX/RoBeY='} 2024-03-25T18:07:01.436 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621321', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:01:38.260293', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB6T5MuIscewO93LAeMlMw2BXb8rpyN4fq4Q2BO2Ncz/pFnpvRkgRfFEX/tARk77Vj9MbOnGxQm1YbS2Y7/PybQ='} 2024-03-25T18:07:01.436 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T18:07:01.442 INFO:teuthology.task.internal:roles: ubuntu@smithi044.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T18:07:01.442 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T18:07:01.443 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T18:07:01.506 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fccaa9620d0>, signals=[15]) 2024-03-25T18:07:01.506 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T18:07:01.513 INFO:teuthology.task.internal:Opening connections... 2024-03-25T18:07:01.513 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi044.front.sepia.ceph.com 2024-03-25T18:07:01.515 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:07:01.618 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi111.front.sepia.ceph.com 2024-03-25T18:07:01.622 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:07:01.707 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T18:07:01.713 DEBUG:teuthology.orchestra.run.smithi044:> uname -m 2024-03-25T18:07:01.730 INFO:teuthology.orchestra.run.smithi044.stdout:x86_64 2024-03-25T18:07:01.731 DEBUG:teuthology.orchestra.run.smithi044:> cat /etc/os-release 2024-03-25T18:07:01.788 INFO:teuthology.orchestra.run.smithi044.stdout:NAME="CentOS Stream" 2024-03-25T18:07:01.788 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION="9" 2024-03-25T18:07:01.788 INFO:teuthology.orchestra.run.smithi044.stdout:ID="centos" 2024-03-25T18:07:01.788 INFO:teuthology.orchestra.run.smithi044.stdout:ID_LIKE="rhel fedora" 2024-03-25T18:07:01.788 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION_ID="9" 2024-03-25T18:07:01.788 INFO:teuthology.orchestra.run.smithi044.stdout:PLATFORM_ID="platform:el9" 2024-03-25T18:07:01.788 INFO:teuthology.orchestra.run.smithi044.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T18:07:01.788 INFO:teuthology.orchestra.run.smithi044.stdout:ANSI_COLOR="0;31" 2024-03-25T18:07:01.789 INFO:teuthology.orchestra.run.smithi044.stdout:LOGO="fedora-logo-icon" 2024-03-25T18:07:01.789 INFO:teuthology.orchestra.run.smithi044.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T18:07:01.789 INFO:teuthology.orchestra.run.smithi044.stdout:HOME_URL="https://centos.org/" 2024-03-25T18:07:01.789 INFO:teuthology.orchestra.run.smithi044.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T18:07:01.789 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T18:07:01.789 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T18:07:01.789 INFO:teuthology.lock.ops:Updating smithi044.front.sepia.ceph.com on lock server 2024-03-25T18:07:01.806 DEBUG:teuthology.orchestra.run.smithi111:> uname -m 2024-03-25T18:07:01.824 INFO:teuthology.orchestra.run.smithi111.stdout:x86_64 2024-03-25T18:07:01.824 DEBUG:teuthology.orchestra.run.smithi111:> cat /etc/os-release 2024-03-25T18:07:01.880 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="CentOS Stream" 2024-03-25T18:07:01.880 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="9" 2024-03-25T18:07:01.880 INFO:teuthology.orchestra.run.smithi111.stdout:ID="centos" 2024-03-25T18:07:01.881 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE="rhel fedora" 2024-03-25T18:07:01.881 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="9" 2024-03-25T18:07:01.881 INFO:teuthology.orchestra.run.smithi111.stdout:PLATFORM_ID="platform:el9" 2024-03-25T18:07:01.881 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T18:07:01.881 INFO:teuthology.orchestra.run.smithi111.stdout:ANSI_COLOR="0;31" 2024-03-25T18:07:01.881 INFO:teuthology.orchestra.run.smithi111.stdout:LOGO="fedora-logo-icon" 2024-03-25T18:07:01.881 INFO:teuthology.orchestra.run.smithi111.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T18:07:01.881 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://centos.org/" 2024-03-25T18:07:01.881 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T18:07:01.881 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T18:07:01.881 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T18:07:01.881 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server 2024-03-25T18:07:01.898 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T18:07:01.907 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T18:07:01.912 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T18:07:01.912 DEBUG:teuthology.orchestra.run.smithi044:> test '!' -e /home/ubuntu/cephtest 2024-03-25T18:07:01.914 DEBUG:teuthology.orchestra.run.smithi111:> test '!' -e /home/ubuntu/cephtest 2024-03-25T18:07:01.936 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T18:07:01.943 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T18:07:01.943 DEBUG:teuthology.orchestra.run.smithi044:> test -z $(ls -A /var/lib/ceph) 2024-03-25T18:07:01.972 DEBUG:teuthology.orchestra.run.smithi111:> test -z $(ls -A /var/lib/ceph) 2024-03-25T18:07:02.000 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T18:07:02.051 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T18:07:02.065 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T18:07:02.065 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T18:07:02.065 DEBUG:teuthology.orchestra.run.smithi044:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T18:07:02.066 DEBUG:teuthology.orchestra.run.smithi111:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T18:07:02.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:07:02.082 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2024-03-25T18:07:02.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:07:02.083 DEBUG:teuthology.orchestra.run.smithi044:> uname -r 2024-03-25T18:07:02.139 INFO:teuthology.orchestra.run.smithi111.stdout:5.14.0-361.el9.x86_64 2024-03-25T18:07:02.139 INFO:teuthology.task.kernel:Running kernel on smithi111: 5.14.0-361.el9.x86_64 2024-03-25T18:07:02.139 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum install -y kernel 2024-03-25T18:07:02.141 INFO:teuthology.orchestra.run.smithi044.stdout:5.14.0-361.el9.x86_64 2024-03-25T18:07:02.141 INFO:teuthology.task.kernel:Running kernel on smithi044: 5.14.0-361.el9.x86_64 2024-03-25T18:07:02.142 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum install -y kernel 2024-03-25T18:07:04.288 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-25T18:07:04.473 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS-9 - AppStream 9.7 MB/s | 6.7 MB 00:00 2024-03-25T18:07:05.788 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS-9 - BaseOS 4.9 MB/s | 2.0 MB 00:00 2024-03-25T18:07:05.932 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-25T18:07:15.294 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - BaseOS 897 kB/s | 8.0 MB 00:09 2024-03-25T18:07:16.426 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - BaseOS 812 kB/s | 8.0 MB 00:10 2024-03-25T18:07:29.305 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - AppStream 1.6 MB/s | 19 MB 00:11 2024-03-25T18:07:41.801 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - CRB 709 kB/s | 6.1 MB 00:08 2024-03-25T18:07:42.741 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - AppStream 732 kB/s | 19 MB 00:26 2024-03-25T18:07:43.385 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-03-25T18:07:45.211 INFO:teuthology.orchestra.run.smithi044.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-25T18:07:50.320 INFO:teuthology.orchestra.run.smithi044.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-25T18:07:52.610 INFO:teuthology.orchestra.run.smithi044.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T18:07:52.610 INFO:teuthology.orchestra.run.smithi044.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T18:07:52.711 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:07:52.718 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:07:52.718 INFO:teuthology.orchestra.run.smithi044.stdout: Package Architecture Version Repository Size 2024-03-25T18:07:52.718 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:07:52.718 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T18:07:52.718 INFO:teuthology.orchestra.run.smithi044.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T18:07:52.719 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-03-25T18:07:52.719 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T18:07:52.719 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T18:07:52.719 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T18:07:52.719 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:07:52.719 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T18:07:52.719 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:07:52.719 INFO:teuthology.orchestra.run.smithi044.stdout:Install 4 Packages 2024-03-25T18:07:52.719 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:07:52.720 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 76 M 2024-03-25T18:07:52.720 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 123 M 2024-03-25T18:07:52.720 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T18:07:53.513 INFO:teuthology.orchestra.run.smithi044.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 91 kB/s | 66 kB 00:00 2024-03-25T18:07:57.752 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - CRB 550 kB/s | 6.1 MB 00:11 2024-03-25T18:07:59.231 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 16 kB 00:00 2024-03-25T18:08:01.926 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 8.0 MB/s | 21 MB 00:02 2024-03-25T18:08:07.023 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 6.4 kB/s | 1.7 kB 00:00 2024-03-25T18:08:09.180 INFO:teuthology.orchestra.run.smithi111.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T18:08:09.180 INFO:teuthology.orchestra.run.smithi111.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T18:08:09.270 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:08:09.282 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:08:09.282 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-03-25T18:08:09.282 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:08:09.283 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T18:08:09.283 INFO:teuthology.orchestra.run.smithi111.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T18:08:09.283 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-25T18:08:09.283 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T18:08:09.283 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T18:08:09.283 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T18:08:09.283 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:08:09.283 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T18:08:09.283 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:08:09.283 INFO:teuthology.orchestra.run.smithi111.stdout:Install 4 Packages 2024-03-25T18:08:09.283 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:08:09.283 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 76 M 2024-03-25T18:08:09.283 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 123 M 2024-03-25T18:08:09.284 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T18:08:09.622 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:08:09.622 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:08:09.622 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:08:09.761 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:08:09.776 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:08:09.800 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:08:09.828 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:08:09.840 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:08:09.909 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:08:09.967 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:08:09.998 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:08:10.033 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:08:10.125 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:08:10.157 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:08:10.192 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:08:10.472 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:08:10.504 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:08:10.638 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:08:10.669 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:08:10.669 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:08:10.689 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:08:10.711 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:08:10.758 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:08:10.839 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:08:10.864 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:08:10.869 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:08:10.902 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:08:10.908 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:08:10.944 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:08:10.953 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:08:10.958 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:08:10.972 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:08:10.980 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:08:11.014 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:08:11.014 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:08:11.047 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:08:11.048 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:08:11.066 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:08:11.066 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:08:11.066 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:08:11.099 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:08:11.118 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:08:11.135 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:08:11.186 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:08:11.204 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:08:11.741 INFO:teuthology.orchestra.run.smithi111.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 30 kB/s | 66 kB 00:02 2024-03-25T18:08:11.845 INFO:teuthology.orchestra.run.smithi044.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 802 kB/s | 15 MB 00:19 2024-03-25T18:08:11.868 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:08:11.937 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:08:12.061 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:08:12.127 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:08:12.478 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:08:12.636 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:08:12.655 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:08:12.692 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:08:12.714 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:08:12.734 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:08:12.788 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:08:12.805 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:08:12.858 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:08:12.875 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:08:22.577 INFO:teuthology.orchestra.run.smithi044.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 968 kB/s | 27 MB 00:29 2024-03-25T18:08:22.978 INFO:teuthology.orchestra.run.smithi044.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 1.1 MB/s | 33 MB 00:30 2024-03-25T18:08:22.978 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:22.978 INFO:teuthology.orchestra.run.smithi044.stdout:Total 2.5 MB/s | 76 MB 00:30 2024-03-25T18:08:23.671 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T18:08:24.104 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T18:08:24.104 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T18:08:25.135 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T18:08:25.135 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T18:08:27.744 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T18:08:28.380 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:08:28.478 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:08:30.714 INFO:teuthology.orchestra.run.smithi111.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 722 kB/s | 15 MB 00:21 2024-03-25T18:08:31.376 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:08:31.507 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:08:36.581 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:08:36.624 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:08:41.721 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:08:48.329 INFO:teuthology.orchestra.run.smithi111.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 769 kB/s | 27 MB 00:36 2024-03-25T18:08:53.687 INFO:teuthology.orchestra.run.smithi111.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 777 kB/s | 33 MB 00:44 2024-03-25T18:08:53.691 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:53.691 INFO:teuthology.orchestra.run.smithi111.stdout:Total 1.7 MB/s | 76 MB 00:44 2024-03-25T18:08:54.358 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T18:08:54.707 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T18:08:54.707 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T18:08:55.699 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T18:08:55.699 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T18:08:58.168 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T18:08:58.786 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:08:58.879 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:09:01.625 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:09:01.756 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:09:06.440 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:09:06.472 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:09:11.211 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:09:12.615 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:09:12.615 INFO:teuthology.orchestra.run.smithi044.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T18:09:12.615 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:09:12.661 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:09:13.051 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:09:13.051 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:09:13.051 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:09:13.051 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:09:13.436 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:09:13.436 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:09:13.436 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T18:09:13.436 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T18:09:13.436 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T18:09:13.436 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T18:09:13.436 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T18:09:13.437 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:09:13.437 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:09:13.770 DEBUG:teuthology.orchestra.run.smithi044:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:09:13.875 INFO:teuthology.orchestra.run.smithi044.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:09:13.877 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:09:13.877 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T18:09:13.877 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T18:09:13.877 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T18:09:13.877 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T18:09:13.877 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi044.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T18:09:13.877 DEBUG:teuthology.orchestra.run.smithi044:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:09:13.960 INFO:teuthology.orchestra.run.smithi044.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:09:13.960 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:09:13.960 DEBUG:teuthology.orchestra.run.smithi044:> sudo rpm -qi grub2-tools 2024-03-25T18:09:13.997 INFO:teuthology.orchestra.run.smithi044.stdout:Name : grub2-tools 2024-03-25T18:09:13.997 INFO:teuthology.orchestra.run.smithi044.stdout:Epoch : 1 2024-03-25T18:09:13.997 INFO:teuthology.orchestra.run.smithi044.stdout:Version : 2.06 2024-03-25T18:09:13.997 INFO:teuthology.orchestra.run.smithi044.stdout:Release : 68.el9 2024-03-25T18:09:13.997 INFO:teuthology.orchestra.run.smithi044.stdout:Architecture: x86_64 2024-03-25T18:09:13.998 INFO:teuthology.orchestra.run.smithi044.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T18:09:13.998 INFO:teuthology.orchestra.run.smithi044.stdout:Group : Unspecified 2024-03-25T18:09:13.998 INFO:teuthology.orchestra.run.smithi044.stdout:Size : 8264254 2024-03-25T18:09:13.998 INFO:teuthology.orchestra.run.smithi044.stdout:License : GPLv3+ 2024-03-25T18:09:13.998 INFO:teuthology.orchestra.run.smithi044.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T18:09:13.998 INFO:teuthology.orchestra.run.smithi044.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T18:09:13.998 INFO:teuthology.orchestra.run.smithi044.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T18:09:13.998 INFO:teuthology.orchestra.run.smithi044.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T18:09:13.998 INFO:teuthology.orchestra.run.smithi044.stdout:Packager : builder@centos.org 2024-03-25T18:09:13.998 INFO:teuthology.orchestra.run.smithi044.stdout:Vendor : CentOS 2024-03-25T18:09:13.998 INFO:teuthology.orchestra.run.smithi044.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T18:09:13.998 INFO:teuthology.orchestra.run.smithi044.stdout:Summary : Support tools for GRUB. 2024-03-25T18:09:13.998 INFO:teuthology.orchestra.run.smithi044.stdout:Description : 2024-03-25T18:09:13.998 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:09:13.999 INFO:teuthology.orchestra.run.smithi044.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T18:09:13.999 INFO:teuthology.orchestra.run.smithi044.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T18:09:13.999 INFO:teuthology.orchestra.run.smithi044.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T18:09:13.999 INFO:teuthology.orchestra.run.smithi044.stdout:hardware devices. 2024-03-25T18:09:13.999 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:09:13.999 INFO:teuthology.orchestra.run.smithi044.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T18:09:14.000 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T18:09:14.000 INFO:teuthology.task.kernel:Updating grub on smithi044 to boot 5.14.0-432.el9.x86_64 2024-03-25T18:09:14.000 DEBUG:teuthology.orchestra.run.smithi044:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T18:09:14.592 INFO:teuthology.orchestra.run.smithi044.stderr:Generating grub configuration file ... 2024-03-25T18:09:15.941 INFO:teuthology.orchestra.run.smithi044.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T18:09:15.968 INFO:teuthology.orchestra.run.smithi044.stderr:done 2024-03-25T18:09:15.970 DEBUG:teuthology.orchestra.run.smithi044:> mktemp 2024-03-25T18:09:15.986 INFO:teuthology.orchestra.run.smithi044.stdout:/tmp/tmp.uRhbaf7Q44 2024-03-25T18:09:15.986 DEBUG:teuthology.orchestra.run.smithi044:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.uRhbaf7Q44 2024-03-25T18:09:16.058 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0666 /tmp/tmp.uRhbaf7Q44 2024-03-25T18:09:16.196 DEBUG:teuthology.orchestra.remote:smithi044:/tmp/tmp.uRhbaf7Q44 is 6KB 2024-03-25T18:09:16.243 DEBUG:teuthology.orchestra.run.smithi044:> rm -fr /tmp/tmp.uRhbaf7Q44 2024-03-25T18:09:16.258 DEBUG:teuthology.orchestra.run.smithi044:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T18:09:16.326 INFO:teuthology.orchestra.run.smithi044.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T18:09:16.326 INFO:teuthology.orchestra.run.smithi044.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T18:09:16.326 INFO:teuthology.orchestra.run.smithi044.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T18:09:16.326 INFO:teuthology.orchestra.run.smithi044.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T18:09:16.327 DEBUG:teuthology.orchestra.run.smithi044:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T18:09:16.497 DEBUG:teuthology.orchestra.run.smithi044:> sudo shutdown -r now 2024-03-25T18:09:40.479 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:09:40.479 INFO:teuthology.orchestra.run.smithi111.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T18:09:40.479 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:09:40.523 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:09:40.925 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:09:40.925 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:09:40.926 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:09:40.926 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:09:41.352 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:09:41.352 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:09:41.352 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T18:09:41.352 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T18:09:41.352 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T18:09:41.352 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T18:09:41.352 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T18:09:41.352 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:09:41.352 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:09:41.719 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:09:41.824 INFO:teuthology.orchestra.run.smithi111.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:09:41.824 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:09:41.824 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T18:09:41.824 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T18:09:41.824 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T18:09:41.825 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T18:09:41.825 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi111.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T18:09:41.825 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:09:41.903 INFO:teuthology.orchestra.run.smithi111.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:09:41.904 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:09:41.904 DEBUG:teuthology.orchestra.run.smithi111:> sudo rpm -qi grub2-tools 2024-03-25T18:09:41.948 INFO:teuthology.orchestra.run.smithi111.stdout:Name : grub2-tools 2024-03-25T18:09:41.949 INFO:teuthology.orchestra.run.smithi111.stdout:Epoch : 1 2024-03-25T18:09:41.949 INFO:teuthology.orchestra.run.smithi111.stdout:Version : 2.06 2024-03-25T18:09:41.949 INFO:teuthology.orchestra.run.smithi111.stdout:Release : 68.el9 2024-03-25T18:09:41.949 INFO:teuthology.orchestra.run.smithi111.stdout:Architecture: x86_64 2024-03-25T18:09:41.949 INFO:teuthology.orchestra.run.smithi111.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T18:09:41.949 INFO:teuthology.orchestra.run.smithi111.stdout:Group : Unspecified 2024-03-25T18:09:41.949 INFO:teuthology.orchestra.run.smithi111.stdout:Size : 8264254 2024-03-25T18:09:41.949 INFO:teuthology.orchestra.run.smithi111.stdout:License : GPLv3+ 2024-03-25T18:09:41.949 INFO:teuthology.orchestra.run.smithi111.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T18:09:41.949 INFO:teuthology.orchestra.run.smithi111.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T18:09:41.949 INFO:teuthology.orchestra.run.smithi111.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T18:09:41.949 INFO:teuthology.orchestra.run.smithi111.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T18:09:41.949 INFO:teuthology.orchestra.run.smithi111.stdout:Packager : builder@centos.org 2024-03-25T18:09:41.950 INFO:teuthology.orchestra.run.smithi111.stdout:Vendor : CentOS 2024-03-25T18:09:41.950 INFO:teuthology.orchestra.run.smithi111.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T18:09:41.950 INFO:teuthology.orchestra.run.smithi111.stdout:Summary : Support tools for GRUB. 2024-03-25T18:09:41.950 INFO:teuthology.orchestra.run.smithi111.stdout:Description : 2024-03-25T18:09:41.950 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:09:41.950 INFO:teuthology.orchestra.run.smithi111.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T18:09:41.950 INFO:teuthology.orchestra.run.smithi111.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T18:09:41.950 INFO:teuthology.orchestra.run.smithi111.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T18:09:41.950 INFO:teuthology.orchestra.run.smithi111.stdout:hardware devices. 2024-03-25T18:09:41.950 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:09:41.950 INFO:teuthology.orchestra.run.smithi111.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T18:09:41.951 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T18:09:41.951 INFO:teuthology.task.kernel:Updating grub on smithi111 to boot 5.14.0-432.el9.x86_64 2024-03-25T18:09:41.951 DEBUG:teuthology.orchestra.run.smithi111:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T18:09:42.538 INFO:teuthology.orchestra.run.smithi111.stderr:Generating grub configuration file ... 2024-03-25T18:09:43.910 INFO:teuthology.orchestra.run.smithi111.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T18:09:43.950 INFO:teuthology.orchestra.run.smithi111.stderr:done 2024-03-25T18:09:43.952 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2024-03-25T18:09:43.968 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.M6DL3tdOCP 2024-03-25T18:09:43.968 DEBUG:teuthology.orchestra.run.smithi111:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.M6DL3tdOCP 2024-03-25T18:09:44.036 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.M6DL3tdOCP 2024-03-25T18:09:44.171 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.M6DL3tdOCP is 6KB 2024-03-25T18:09:44.220 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.M6DL3tdOCP 2024-03-25T18:09:44.233 DEBUG:teuthology.orchestra.run.smithi111:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T18:09:44.305 INFO:teuthology.orchestra.run.smithi111.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T18:09:44.306 INFO:teuthology.orchestra.run.smithi111.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T18:09:44.306 INFO:teuthology.orchestra.run.smithi111.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T18:09:44.306 INFO:teuthology.orchestra.run.smithi111.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T18:09:44.307 DEBUG:teuthology.orchestra.run.smithi111:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T18:09:44.456 DEBUG:teuthology.orchestra.run.smithi111:> sudo shutdown -r now 2024-03-25T18:09:46.501 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T18:09:46.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-03-25T18:09:46.503 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:10:04.962 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.44 2024-03-25T18:10:13.965 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-03-25T18:10:13.966 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:10:14.458 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T18:10:14.459 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-25T18:10:14.459 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:10:32.862 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-03-25T18:10:41.866 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-25T18:10:41.866 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:10:44.926 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-03-25T18:10:56.938 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-25T18:10:56.939 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:10:59.998 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-03-25T18:11:13.982 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T18:11:15.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-25T18:11:15.001 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:11:25.994 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-03-25T18:11:25.995 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:11:26.352 DEBUG:teuthology.orchestra.run.smithi044:> true 2024-03-25T18:11:26.714 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-03-25T18:11:26.714 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-25T18:11:26.714 DEBUG:teuthology.orchestra.run.smithi044:> uname -r 2024-03-25T18:11:26.729 INFO:teuthology.orchestra.run.smithi044.stdout:5.14.0-432.el9.x86_64 2024-03-25T18:11:26.729 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T18:11:26.729 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T18:11:26.729 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T18:11:27.731 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T18:11:27.731 DEBUG:teuthology.orchestra.run.smithi044:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T18:11:27.811 INFO:teuthology.orchestra.run.smithi044.stdout:ttyS1 2024-03-25T18:11:27.848 DEBUG:teuthology.parallel:result is None 2024-03-25T18:11:46.957 DEBUG:teuthology.orchestra.run.smithi111:> true 2024-03-25T18:11:47.486 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-25T18:11:47.486 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-25T18:11:47.486 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2024-03-25T18:11:47.513 INFO:teuthology.orchestra.run.smithi111.stdout:5.14.0-432.el9.x86_64 2024-03-25T18:11:47.513 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T18:11:47.513 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T18:11:47.513 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T18:11:48.514 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T18:11:48.514 DEBUG:teuthology.orchestra.run.smithi111:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T18:11:48.574 INFO:teuthology.orchestra.run.smithi111.stdout:ttyS1 2024-03-25T18:11:48.747 DEBUG:teuthology.parallel:result is None 2024-03-25T18:11:48.747 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T18:11:48.756 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T18:11:48.757 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T18:11:48.759 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T18:11:48.774 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T18:11:48.786 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T18:11:48.800 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T18:11:48.800 DEBUG:teuthology.orchestra.run.smithi044:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T18:11:48.817 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T18:11:48.847 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T18:11:48.858 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T18:11:48.859 DEBUG:teuthology.orchestra.run.smithi044:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T18:11:48.883 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T18:11:48.914 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:11:48.917 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:11:48.927 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:11:48.932 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:11:48.933 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T18:11:48.945 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T18:11:48.945 DEBUG:teuthology.orchestra.run.smithi044:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T18:11:48.976 DEBUG:teuthology.orchestra.run.smithi111:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T18:11:49.003 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T18:11:49.018 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T18:11:49.018 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T18:11:49.045 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T18:11:49.059 DEBUG:teuthology.orchestra.run.smithi044:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:11:49.139 DEBUG:teuthology.orchestra.run.smithi044:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:11:49.189 DEBUG:teuthology.orchestra.run.smithi044:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:11:49.269 DEBUG:teuthology.orchestra.run.smithi044:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:11:49.314 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T18:11:49.314 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T18:11:49.382 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:11:49.423 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:11:49.474 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:11:49.556 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:11:49.596 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T18:11:49.596 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T18:11:49.662 DEBUG:teuthology.orchestra.run.smithi044:> sudo service rsyslog restart 2024-03-25T18:11:49.664 DEBUG:teuthology.orchestra.run.smithi111:> sudo service rsyslog restart 2024-03-25T18:11:49.704 INFO:teuthology.orchestra.run.smithi044.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:11:49.753 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:11:50.260 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T18:11:50.269 INFO:teuthology.task.internal:Starting timer... 2024-03-25T18:11:50.270 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T18:11:50.286 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T18:11:50.299 DEBUG:teuthology.orchestra.run.smithi044:> sudo service auditd rotate 2024-03-25T18:11:50.348 INFO:teuthology.orchestra.run.smithi044.stdout:Rotating logs: 2024-03-25T18:11:50.349 DEBUG:teuthology.orchestra.run.smithi111:> sudo service auditd rotate 2024-03-25T18:11:50.392 INFO:teuthology.orchestra.run.smithi111.stdout:Rotating logs: 2024-03-25T18:11:50.393 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T18:11:50.393 DEBUG:teuthology.orchestra.run.smithi044:> /usr/sbin/getenforce 2024-03-25T18:11:50.417 INFO:teuthology.orchestra.run.smithi044.stdout:Permissive 2024-03-25T18:11:50.417 DEBUG:teuthology.orchestra.run.smithi111:> /usr/sbin/getenforce 2024-03-25T18:11:50.464 INFO:teuthology.orchestra.run.smithi111.stdout:Permissive 2024-03-25T18:11:50.464 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi044.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi111.front.sepia.ceph.com': 'permissive'} 2024-03-25T18:11:50.464 DEBUG:teuthology.orchestra.run.smithi044:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:11:50.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:11:50.492 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:11:50.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:11:50.531 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T18:11:50.531 DEBUG:teuthology.orchestra.run.smithi044:> sudo /usr/sbin/setenforce permissive 2024-03-25T18:11:50.569 DEBUG:teuthology.orchestra.run.smithi111:> sudo /usr/sbin/setenforce permissive 2024-03-25T18:11:50.607 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T18:11:50.624 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-25T18:11:50.630 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-25T18:11:50.778 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T18:11:50.793 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T18:11:50.795 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi044.front.sepia.ceph.com,smithi111.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T18:16:29.705 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi044.front.sepia.ceph.com'), Remote(name='ubuntu@smithi111.front.sepia.ceph.com')] 2024-03-25T18:16:29.706 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-03-25T18:16:29.707 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:16:29.796 DEBUG:teuthology.orchestra.run.smithi044:> true 2024-03-25T18:16:29.873 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-03-25T18:16:29.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-25T18:16:29.874 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:16:29.959 DEBUG:teuthology.orchestra.run.smithi111:> true 2024-03-25T18:16:30.038 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-25T18:16:30.038 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T18:16:30.049 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T18:16:30.049 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T18:16:30.050 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:16:30.052 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T18:16:30.052 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:16:30.081 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T18:16:30.098 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T18:16:30.108 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T18:16:30.125 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T18:16:30.127 INFO:teuthology.orchestra.run.smithi044.stderr:sudo: ntpd: command not found 2024-03-25T18:16:30.141 INFO:teuthology.orchestra.run.smithi044.stdout:506 Cannot talk to daemon 2024-03-25T18:16:30.155 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: ntpd: command not found 2024-03-25T18:16:30.157 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T18:16:30.169 INFO:teuthology.orchestra.run.smithi111.stdout:506 Cannot talk to daemon 2024-03-25T18:16:30.174 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T18:16:30.185 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T18:16:30.202 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T18:16:30.230 INFO:teuthology.orchestra.run.smithi044.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:16:30.234 INFO:teuthology.orchestra.run.smithi044.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:16:30.234 INFO:teuthology.orchestra.run.smithi044.stdout:=============================================================================== 2024-03-25T18:16:30.234 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:16:30.234 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:16:30.235 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:16:30.235 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:16:30.255 INFO:teuthology.orchestra.run.smithi111.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:16:30.259 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:16:30.259 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-03-25T18:16:30.259 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:16:30.259 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:16:30.259 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:16:30.260 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:16:30.260 INFO:teuthology.run_tasks:Running task install... 2024-03-25T18:16:30.268 DEBUG:teuthology.task.install:project ceph 2024-03-25T18:16:30.268 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T18:16:30.268 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T18:16:30.268 INFO:teuthology.task.install:Using flavor: default 2024-03-25T18:16:30.273 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T18:16:30.273 INFO:teuthology.task.install:extra packages: [] 2024-03-25T18:16:30.274 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T18:16:30.274 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:16:30.276 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T18:16:30.276 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:16:30.417 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T18:16:30.417 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T18:16:30.497 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T18:16:30.497 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T18:16:30.761 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T18:16:30.762 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T18:16:30.762 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T18:16:30.791 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T18:16:30.791 DEBUG:teuthology.orchestra.run.smithi111:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T18:16:30.860 DEBUG:teuthology.orchestra.run.smithi111:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T18:16:30.925 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T18:16:30.925 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T18:16:30.925 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T18:16:30.940 DEBUG:teuthology.orchestra.run.smithi111:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T18:16:30.955 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T18:16:30.956 DEBUG:teuthology.orchestra.run.smithi044:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T18:16:31.006 INFO:teuthology.orchestra.run.smithi111.stdout:check_obsoletes = 1 2024-03-25T18:16:31.008 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum clean all 2024-03-25T18:16:31.026 DEBUG:teuthology.orchestra.run.smithi044:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T18:16:31.110 DEBUG:teuthology.orchestra.run.smithi044:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T18:16:31.178 INFO:teuthology.orchestra.run.smithi044.stdout:check_obsoletes = 1 2024-03-25T18:16:31.180 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum clean all 2024-03-25T18:16:31.249 INFO:teuthology.orchestra.run.smithi111.stdout:74 files removed 2024-03-25T18:16:31.276 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-radosgw 2024-03-25T18:16:31.430 INFO:teuthology.orchestra.run.smithi044.stdout:74 files removed 2024-03-25T18:16:31.457 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-radosgw 2024-03-25T18:16:32.591 INFO:teuthology.orchestra.run.smithi111.stdout:ceph packages for x86_64 80 kB/s | 82 kB 00:01 2024-03-25T18:16:32.839 INFO:teuthology.orchestra.run.smithi044.stdout:ceph packages for x86_64 76 kB/s | 82 kB 00:01 2024-03-25T18:16:33.500 INFO:teuthology.orchestra.run.smithi111.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T18:16:33.740 INFO:teuthology.orchestra.run.smithi044.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T18:16:34.284 INFO:teuthology.orchestra.run.smithi111.stdout:ceph source packages 2.5 kB/s | 1.9 kB 00:00 2024-03-25T18:16:34.565 INFO:teuthology.orchestra.run.smithi044.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-03-25T18:16:34.995 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-25T18:16:35.017 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-25T18:16:36.500 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2024-03-25T18:16:36.503 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS-9 - BaseOS 5.2 MB/s | 2.0 MB 00:00 2024-03-25T18:16:46.838 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - BaseOS 825 kB/s | 8.0 MB 00:09 2024-03-25T18:18:17.164 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - BaseOS 82 kB/s | 8.0 MB 01:40 2024-03-25T18:18:27.755 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - AppStream 194 kB/s | 19 MB 01:39 2024-03-25T18:18:39.135 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - AppStream 925 kB/s | 19 MB 00:20 2024-03-25T18:18:42.825 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - CRB 551 kB/s | 6.1 MB 00:11 2024-03-25T18:18:44.530 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-03-25T18:18:46.042 INFO:teuthology.orchestra.run.smithi044.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-25T18:18:51.173 INFO:teuthology.orchestra.run.smithi044.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-03-25T18:18:53.697 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - CRB 569 kB/s | 6.1 MB 00:11 2024-03-25T18:18:53.767 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:18:53.775 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:18:53.775 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-03-25T18:18:53.775 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:18:53.775 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T18:18:53.775 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:18:53.775 INFO:teuthology.orchestra.run.smithi044.stdout:Upgrading: 2024-03-25T18:18:53.775 INFO:teuthology.orchestra.run.smithi044.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:18:53.776 INFO:teuthology.orchestra.run.smithi044.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:18:53.776 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-03-25T18:18:53.776 INFO:teuthology.orchestra.run.smithi044.stdout: boost-program-options 2024-03-25T18:18:53.776 INFO:teuthology.orchestra.run.smithi044.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:18:53.776 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:18:53.776 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:18:53.776 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:18:53.776 INFO:teuthology.orchestra.run.smithi044.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:18:53.776 INFO:teuthology.orchestra.run.smithi044.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:18:53.776 INFO:teuthology.orchestra.run.smithi044.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:18:53.776 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:18:53.776 INFO:teuthology.orchestra.run.smithi044.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:18:53.776 INFO:teuthology.orchestra.run.smithi044.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:18:53.776 INFO:teuthology.orchestra.run.smithi044.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:18:53.777 INFO:teuthology.orchestra.run.smithi044.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:18:53.777 INFO:teuthology.orchestra.run.smithi044.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:18:53.777 INFO:teuthology.orchestra.run.smithi044.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:18:53.777 INFO:teuthology.orchestra.run.smithi044.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:18:53.777 INFO:teuthology.orchestra.run.smithi044.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:18:53.777 INFO:teuthology.orchestra.run.smithi044.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:18:53.777 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-argparse 2024-03-25T18:18:53.777 INFO:teuthology.orchestra.run.smithi044.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:18:53.777 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:18:53.777 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:18:53.777 INFO:teuthology.orchestra.run.smithi044.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:18:53.777 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:18:53.778 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:18:53.778 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:18:53.778 INFO:teuthology.orchestra.run.smithi044.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:18:53.778 INFO:teuthology.orchestra.run.smithi044.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:18:53.778 INFO:teuthology.orchestra.run.smithi044.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:18:53.778 INFO:teuthology.orchestra.run.smithi044.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:18:53.778 INFO:teuthology.orchestra.run.smithi044.stdout:Installing weak dependencies: 2024-03-25T18:18:53.778 INFO:teuthology.orchestra.run.smithi044.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:18:53.778 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:18:53.778 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T18:18:53.778 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:18:53.778 INFO:teuthology.orchestra.run.smithi044.stdout:Install 30 Packages 2024-03-25T18:18:53.778 INFO:teuthology.orchestra.run.smithi044.stdout:Upgrade 2 Packages 2024-03-25T18:18:53.778 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:18:53.779 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 58 M 2024-03-25T18:18:53.779 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T18:18:55.260 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - Extras packages 75 kB/s | 16 kB 00:00 2024-03-25T18:19:04.332 INFO:teuthology.orchestra.run.smithi044.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 527 kB/s | 5.3 MB 00:10 2024-03-25T18:19:04.432 INFO:teuthology.orchestra.run.smithi044.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-25T18:19:05.557 INFO:teuthology.orchestra.run.smithi044.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 659 kB/s | 742 kB 00:01 2024-03-25T18:19:06.332 INFO:teuthology.orchestra.run.smithi044.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 643 kB/s | 498 kB 00:00 2024-03-25T18:19:13.524 INFO:teuthology.orchestra.run.smithi044.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 726 kB/s | 5.1 MB 00:07 2024-03-25T18:19:13.640 INFO:teuthology.orchestra.run.smithi044.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 388 kB/s | 45 kB 00:00 2024-03-25T18:19:13.891 INFO:teuthology.orchestra.run.smithi044.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 525 kB/s | 131 kB 00:00 2024-03-25T18:19:14.216 INFO:teuthology.orchestra.run.smithi044.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 498 kB/s | 162 kB 00:00 2024-03-25T18:19:14.950 INFO:teuthology.orchestra.run.smithi044.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 437 kB/s | 320 kB 00:00 2024-03-25T18:19:15.773 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:20 2024-03-25T18:19:15.841 INFO:teuthology.orchestra.run.smithi044.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 334 kB/s | 298 kB 00:00 2024-03-25T18:19:16.267 INFO:teuthology.orchestra.run.smithi044.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 234 kB/s | 99 kB 00:00 2024-03-25T18:19:16.417 INFO:teuthology.orchestra.run.smithi044.stdout:(12/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 444 kB/s | 9.7 MB 00:22 2024-03-25T18:19:16.468 INFO:teuthology.orchestra.run.smithi044.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 528 kB/s | 106 kB 00:00 2024-03-25T18:19:16.492 INFO:teuthology.orchestra.run.smithi044.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 159 kB 00:00 2024-03-25T18:19:16.518 INFO:teuthology.orchestra.run.smithi044.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 887 kB/s | 44 kB 00:00 2024-03-25T18:19:16.551 INFO:teuthology.orchestra.run.smithi044.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T18:19:16.577 INFO:teuthology.orchestra.run.smithi044.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-03-25T18:19:16.602 INFO:teuthology.orchestra.run.smithi044.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 827 kB/s | 41 kB 00:00 2024-03-25T18:19:16.729 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:19:16.746 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:19:16.748 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:19:16.764 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:19:16.910 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:19:16.923 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:19:17.262 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:19:17.294 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:19:17.367 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:19:17.367 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:19:17.500 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:19:17.524 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:19:17.526 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:19:17.563 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:19:17.599 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:19:17.617 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:19:17.640 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:19:17.658 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:19:17.733 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:19:17.758 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:19:17.817 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:19:17.817 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:19:17.909 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:19:17.929 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:19:17.974 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:19:17.994 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:19:18.098 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:19:18.184 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:19:18.711 INFO:teuthology.orchestra.run.smithi044.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 17 kB/s | 36 kB 00:02 2024-03-25T18:19:18.834 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:19:18.855 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:19:19.014 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:19:19.044 INFO:teuthology.orchestra.run.smithi044.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 89 kB/s | 221 kB 00:02 2024-03-25T18:19:19.369 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:19:19.438 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:19:19.571 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:19:19.590 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:19:19.628 INFO:teuthology.orchestra.run.smithi044.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 530 kB/s | 309 kB 00:00 2024-03-25T18:19:19.677 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:19:19.759 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:19:19.914 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:19:19.957 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:19:20.050 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:19:20.117 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:19:20.195 INFO:teuthology.orchestra.run.smithi044.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 8.2 MB/s | 4.4 MB 00:00 2024-03-25T18:19:20.287 INFO:teuthology.orchestra.run.smithi044.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:19:20.328 INFO:teuthology.orchestra.run.smithi044.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 243 kB/s | 26 kB 00:00 2024-03-25T18:19:20.428 INFO:teuthology.orchestra.run.smithi044.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 485 kB/s | 49 kB 00:00 2024-03-25T18:19:20.529 INFO:teuthology.orchestra.run.smithi044.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 673 kB/s | 67 kB 00:00 2024-03-25T18:19:20.669 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-25T18:19:20.687 INFO:teuthology.orchestra.run.smithi044.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.2 MB/s | 838 kB 00:00 2024-03-25T18:19:20.796 INFO:teuthology.orchestra.run.smithi044.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-25T18:19:20.938 INFO:teuthology.orchestra.run.smithi044.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-25T18:19:20.963 INFO:teuthology.orchestra.run.smithi044.stdout:(29/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 21 kB/s | 47 kB 00:02 2024-03-25T18:19:23.112 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:19:23.135 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:19:23.136 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-25T18:19:23.136 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:19:23.136 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T18:19:23.136 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:19:23.136 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrading: 2024-03-25T18:19:23.136 INFO:teuthology.orchestra.run.smithi111.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:19:23.136 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:19:23.136 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-25T18:19:23.136 INFO:teuthology.orchestra.run.smithi111.stdout: boost-program-options 2024-03-25T18:19:23.136 INFO:teuthology.orchestra.run.smithi111.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:19:23.136 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:19:23.136 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:19:23.136 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:19:23.136 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:19:23.136 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:19:23.137 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:19:23.137 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:19:23.137 INFO:teuthology.orchestra.run.smithi111.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:19:23.137 INFO:teuthology.orchestra.run.smithi111.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:19:23.137 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:19:23.137 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:19:23.137 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:19:23.137 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:19:23.137 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:19:23.137 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:19:23.137 INFO:teuthology.orchestra.run.smithi111.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:19:23.137 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse 2024-03-25T18:19:23.137 INFO:teuthology.orchestra.run.smithi111.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:19:23.137 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:19:23.138 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:19:23.138 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:19:23.138 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:19:23.138 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:19:23.138 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:19:23.138 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:19:23.138 INFO:teuthology.orchestra.run.smithi111.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:19:23.138 INFO:teuthology.orchestra.run.smithi111.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:19:23.138 INFO:teuthology.orchestra.run.smithi111.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:19:23.138 INFO:teuthology.orchestra.run.smithi111.stdout:Installing weak dependencies: 2024-03-25T18:19:23.138 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:19:23.138 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:19:23.138 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T18:19:23.138 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:19:23.138 INFO:teuthology.orchestra.run.smithi111.stdout:Install 30 Packages 2024-03-25T18:19:23.138 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrade 2 Packages 2024-03-25T18:19:23.139 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:19:23.139 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 58 M 2024-03-25T18:19:23.139 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T18:19:27.797 INFO:teuthology.orchestra.run.smithi044.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 469 kB/s | 3.1 MB 00:06 2024-03-25T18:19:29.121 INFO:teuthology.orchestra.run.smithi044.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 422 kB/s | 3.4 MB 00:08 2024-03-25T18:19:39.035 INFO:teuthology.orchestra.run.smithi111.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 645 kB/s | 9.7 MB 00:15 2024-03-25T18:19:39.135 INFO:teuthology.orchestra.run.smithi111.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-25T18:19:40.002 INFO:teuthology.orchestra.run.smithi111.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 856 kB/s | 742 kB 00:00 2024-03-25T18:19:40.677 INFO:teuthology.orchestra.run.smithi111.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 739 kB/s | 498 kB 00:00 2024-03-25T18:19:41.628 INFO:teuthology.orchestra.run.smithi111.stdout:(5/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 303 kB/s | 5.3 MB 00:18 2024-03-25T18:19:41.836 INFO:teuthology.orchestra.run.smithi111.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 217 kB/s | 45 kB 00:00 2024-03-25T18:19:42.228 INFO:teuthology.orchestra.run.smithi111.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 335 kB/s | 131 kB 00:00 2024-03-25T18:19:42.645 INFO:teuthology.orchestra.run.smithi111.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 389 kB/s | 162 kB 00:00 2024-03-25T18:19:43.445 INFO:teuthology.orchestra.run.smithi111.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 401 kB/s | 320 kB 00:00 2024-03-25T18:19:44.178 INFO:teuthology.orchestra.run.smithi111.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 406 kB/s | 298 kB 00:00 2024-03-25T18:19:44.479 INFO:teuthology.orchestra.run.smithi111.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 331 kB/s | 99 kB 00:00 2024-03-25T18:19:44.595 INFO:teuthology.orchestra.run.smithi111.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 907 kB/s | 106 kB 00:00 2024-03-25T18:19:44.646 INFO:teuthology.orchestra.run.smithi111.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T18:19:44.679 INFO:teuthology.orchestra.run.smithi111.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T18:19:44.738 INFO:teuthology.orchestra.run.smithi111.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T18:19:44.780 INFO:teuthology.orchestra.run.smithi111.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T18:19:44.822 INFO:teuthology.orchestra.run.smithi111.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 984 kB/s | 41 kB 00:00 2024-03-25T18:19:44.873 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:19:44.953 INFO:teuthology.orchestra.run.smithi044.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 399 kB/s | 20 MB 00:51 2024-03-25T18:19:44.960 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:19:44.960 INFO:teuthology.orchestra.run.smithi044.stdout:Total 1.1 MB/s | 58 MB 00:51 2024-03-25T18:19:44.971 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:19:44.991 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T18:19:45.032 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T18:19:45.032 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T18:19:45.040 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:19:45.167 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:19:45.244 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:19:45.388 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:19:45.519 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T18:19:45.519 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T18:19:45.521 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:19:45.540 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:19:45.627 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:19:45.770 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:19:45.805 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:19:46.689 INFO:teuthology.orchestra.run.smithi111.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 118 kB/s | 221 kB 00:01 2024-03-25T18:19:46.734 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:19:46.837 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:19:46.914 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:19:47.001 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T18:19:47.057 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:19:47.077 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:19:47.108 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:19:47.126 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:19:47.134 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:19:47.156 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:19:47.173 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:19:47.262 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:19:47.357 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:19:47.383 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:19:47.401 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:19:47.412 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:19:47.477 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:19:47.647 INFO:teuthology.orchestra.run.smithi111.stdout:(19/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 749 kB/s | 5.1 MB 00:06 2024-03-25T18:19:47.648 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:19:47.677 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:19:47.682 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:19:47.768 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:19:47.802 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:19:47.835 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:19:47.837 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:19:47.928 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:19:47.962 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:19:48.029 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:19:48.045 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:19:48.079 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:19:48.142 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:19:48.163 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:19:48.171 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:19:48.215 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:19:48.281 INFO:teuthology.orchestra.run.smithi111.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 23 kB/s | 36 kB 00:01 2024-03-25T18:19:48.284 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:19:48.301 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:19:48.307 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:19:48.321 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:19:48.362 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:19:48.396 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:19:48.400 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:19:48.437 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:19:48.454 INFO:teuthology.orchestra.run.smithi111.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:19:48.507 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:19:48.531 INFO:teuthology.orchestra.run.smithi111.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-03-25T18:19:48.556 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:19:48.596 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:19:48.629 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:19:48.658 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:19:48.691 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:19:49.013 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:19:49.046 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:19:49.098 INFO:teuthology.orchestra.run.smithi111.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 33 kB/s | 47 kB 00:01 2024-03-25T18:19:49.120 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:19:49.151 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:19:49.190 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:19:49.298 INFO:teuthology.orchestra.run.smithi111.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 130 kB/s | 26 kB 00:00 2024-03-25T18:19:49.399 INFO:teuthology.orchestra.run.smithi111.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 486 kB/s | 49 kB 00:00 2024-03-25T18:19:49.508 INFO:teuthology.orchestra.run.smithi111.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.5 MB/s | 4.4 MB 00:00 2024-03-25T18:19:49.533 INFO:teuthology.orchestra.run.smithi111.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 504 kB/s | 67 kB 00:00 2024-03-25T18:19:49.624 INFO:teuthology.orchestra.run.smithi111.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-03-25T18:19:49.666 INFO:teuthology.orchestra.run.smithi111.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.2 MB/s | 838 kB 00:00 2024-03-25T18:19:49.667 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:19:49.699 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:19:49.753 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:19:49.824 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:19:49.859 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:19:50.092 INFO:teuthology.orchestra.run.smithi111.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 3.4 MB/s | 1.6 MB 00:00 2024-03-25T18:19:52.158 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:19:52.198 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:19:52.778 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:19:52.818 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:19:52.908 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:19:52.908 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:19:52.909 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:19:52.909 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:19:52.953 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:19:53.608 INFO:teuthology.orchestra.run.smithi111.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 911 kB/s | 3.1 MB 00:03 2024-03-25T18:19:53.884 INFO:teuthology.orchestra.run.smithi111.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 819 kB/s | 3.4 MB 00:04 2024-03-25T18:20:02.776 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:20:02.777 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /sys 2024-03-25T18:20:02.777 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /proc 2024-03-25T18:20:02.777 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /mnt 2024-03-25T18:20:02.777 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /var/tmp 2024-03-25T18:20:02.777 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /home 2024-03-25T18:20:02.777 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /root 2024-03-25T18:20:02.777 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /tmp 2024-03-25T18:20:02.777 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:20:03.709 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:20:03.733 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:20:03.733 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:20:03.733 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:20:03.733 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:20:03.733 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:20:03.733 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:20:03.736 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:20:03.812 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:20:03.812 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:20:04.991 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:20:04.991 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:20:04.991 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:20:04.991 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:20:04.991 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:20:04.991 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:20:04.991 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:20:04.991 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:20:04.991 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:20:04.991 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:20:04.991 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:20:04.992 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:20:04.992 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:20:04.992 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:20:04.992 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:20:04.992 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:20:04.992 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:20:04.992 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:20:04.992 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:20:04.992 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:20:04.992 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:20:04.992 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:20:04.992 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:20:04.992 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:20:04.992 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:20:04.992 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:20:04.993 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:20:04.993 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:20:04.993 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:20:04.993 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:20:04.993 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:20:04.993 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:20:04.994 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:20:04.994 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:20:06.248 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:20:06.248 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:20:06.248 INFO:teuthology.orchestra.run.smithi044.stdout:Upgraded: 2024-03-25T18:20:06.248 INFO:teuthology.orchestra.run.smithi044.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:06.248 INFO:teuthology.orchestra.run.smithi044.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:06.248 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T18:20:06.249 INFO:teuthology.orchestra.run.smithi044.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:20:06.249 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:06.249 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:06.249 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:06.249 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:06.249 INFO:teuthology.orchestra.run.smithi044.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:20:06.249 INFO:teuthology.orchestra.run.smithi044.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:20:06.249 INFO:teuthology.orchestra.run.smithi044.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:20:06.249 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:06.249 INFO:teuthology.orchestra.run.smithi044.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:20:06.249 INFO:teuthology.orchestra.run.smithi044.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:20:06.249 INFO:teuthology.orchestra.run.smithi044.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:20:06.249 INFO:teuthology.orchestra.run.smithi044.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:06.250 INFO:teuthology.orchestra.run.smithi044.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:20:06.250 INFO:teuthology.orchestra.run.smithi044.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:06.250 INFO:teuthology.orchestra.run.smithi044.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:20:06.250 INFO:teuthology.orchestra.run.smithi044.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:20:06.250 INFO:teuthology.orchestra.run.smithi044.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:20:06.250 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:06.250 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:06.250 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:06.250 INFO:teuthology.orchestra.run.smithi044.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:20:06.250 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:06.250 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:06.250 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:06.250 INFO:teuthology.orchestra.run.smithi044.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:20:06.250 INFO:teuthology.orchestra.run.smithi044.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:20:06.251 INFO:teuthology.orchestra.run.smithi044.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:20:06.251 INFO:teuthology.orchestra.run.smithi044.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:20:06.251 INFO:teuthology.orchestra.run.smithi044.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:20:06.251 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:20:06.251 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:20:06.515 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-test 2024-03-25T18:20:07.085 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:01:16 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:20:07.240 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:20:07.241 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:20:07.241 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-03-25T18:20:07.241 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:20:07.241 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T18:20:07.241 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:20:07.241 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-03-25T18:20:07.241 INFO:teuthology.orchestra.run.smithi044.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:20:07.241 INFO:teuthology.orchestra.run.smithi044.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:20:07.241 INFO:teuthology.orchestra.run.smithi044.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:20:07.241 INFO:teuthology.orchestra.run.smithi044.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:20:07.241 INFO:teuthology.orchestra.run.smithi044.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:20:07.241 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:20:07.242 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T18:20:07.242 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:20:07.242 INFO:teuthology.orchestra.run.smithi044.stdout:Install 6 Packages 2024-03-25T18:20:07.242 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:20:07.242 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 48 M 2024-03-25T18:20:07.242 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 204 M 2024-03-25T18:20:07.242 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T18:20:07.634 INFO:teuthology.orchestra.run.smithi044.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.8 MB/s | 217 kB 00:00 2024-03-25T18:20:07.750 INFO:teuthology.orchestra.run.smithi044.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.3 MB/s | 300 kB 00:00 2024-03-25T18:20:07.793 INFO:teuthology.orchestra.run.smithi044.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.4 MB/s | 63 kB 00:00 2024-03-25T18:20:07.935 INFO:teuthology.orchestra.run.smithi044.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-25T18:20:08.377 INFO:teuthology.orchestra.run.smithi044.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 189 kB/s | 163 kB 00:00 2024-03-25T18:20:16.864 INFO:teuthology.orchestra.run.smithi111.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 382 kB/s | 20 MB 00:53 2024-03-25T18:20:16.872 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:20:16.872 INFO:teuthology.orchestra.run.smithi111.stdout:Total 1.1 MB/s | 58 MB 00:53 2024-03-25T18:20:16.905 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T18:20:16.944 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T18:20:16.944 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T18:20:17.426 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T18:20:17.426 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T18:20:18.934 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T18:20:19.020 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:20:19.052 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:20:19.078 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:20:19.100 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:20:19.117 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:20:19.309 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:20:19.365 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:20:19.717 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:20:19.779 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:20:19.871 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:20:19.960 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:20:19.989 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:20:20.052 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:20:20.080 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:20:20.124 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:20:20.192 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:20:20.216 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:20:20.263 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:20:20.299 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:20:20.405 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:20:20.450 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:20:20.490 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:20:20.522 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:20:20.551 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:20:20.584 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:20:20.899 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:20:20.931 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:20:21.020 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:20:21.052 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:20:21.099 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:20:21.572 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:20:21.600 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:20:21.671 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:20:21.748 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:20:21.778 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:20:23.917 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:20:23.966 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:20:24.524 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:20:24.561 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:20:24.636 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:20:24.636 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:20:24.636 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:20:24.636 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:20:24.679 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:20:33.910 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:20:33.911 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /sys 2024-03-25T18:20:33.911 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /proc 2024-03-25T18:20:33.911 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /mnt 2024-03-25T18:20:33.911 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /var/tmp 2024-03-25T18:20:33.911 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /home 2024-03-25T18:20:33.911 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /root 2024-03-25T18:20:33.911 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /tmp 2024-03-25T18:20:33.911 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:20:34.831 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:20:34.855 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:20:34.855 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:20:34.855 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:20:34.855 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:20:34.855 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:20:34.856 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:20:34.857 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:20:34.942 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:20:34.943 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:20:36.063 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:20:36.063 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:20:36.063 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:20:36.063 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:20:36.063 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:20:36.063 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:20:36.064 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:20:36.064 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:20:36.064 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:20:36.064 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:20:36.064 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:20:36.064 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:20:36.064 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:20:36.064 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:20:36.064 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:20:36.064 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:20:36.064 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:20:36.064 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:20:36.064 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:20:36.064 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:20:36.065 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:20:36.065 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:20:36.065 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:20:36.065 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:20:36.065 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:20:36.065 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:20:36.065 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:20:36.065 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:20:36.065 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:20:36.065 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:20:36.065 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:20:36.066 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:20:36.066 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:20:36.066 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:20:37.058 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:20:37.058 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:20:37.059 INFO:teuthology.orchestra.run.smithi111.stdout:Upgraded: 2024-03-25T18:20:37.059 INFO:teuthology.orchestra.run.smithi111.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:37.059 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:37.059 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T18:20:37.059 INFO:teuthology.orchestra.run.smithi111.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:20:37.059 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:37.059 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:37.059 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:37.059 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:37.059 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:20:37.059 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:20:37.059 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:20:37.059 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:37.059 INFO:teuthology.orchestra.run.smithi111.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:20:37.060 INFO:teuthology.orchestra.run.smithi111.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:20:37.060 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:20:37.060 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:37.060 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:20:37.060 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:37.060 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:20:37.060 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:20:37.060 INFO:teuthology.orchestra.run.smithi111.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:20:37.060 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:37.060 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:37.060 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:37.060 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:20:37.060 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:37.060 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:37.061 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:37.061 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:20:37.061 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:20:37.061 INFO:teuthology.orchestra.run.smithi111.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:20:37.061 INFO:teuthology.orchestra.run.smithi111.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:20:37.061 INFO:teuthology.orchestra.run.smithi111.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:20:37.061 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:20:37.061 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:20:37.404 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-test 2024-03-25T18:20:37.963 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:17 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:20:38.113 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:20:38.114 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:20:38.114 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-25T18:20:38.114 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:20:38.114 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T18:20:38.114 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:20:38.115 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-25T18:20:38.115 INFO:teuthology.orchestra.run.smithi111.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:20:38.115 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:20:38.115 INFO:teuthology.orchestra.run.smithi111.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:20:38.115 INFO:teuthology.orchestra.run.smithi111.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:20:38.115 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:20:38.115 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:20:38.115 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T18:20:38.115 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:20:38.115 INFO:teuthology.orchestra.run.smithi111.stdout:Install 6 Packages 2024-03-25T18:20:38.115 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:20:38.116 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 48 M 2024-03-25T18:20:38.116 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 204 M 2024-03-25T18:20:38.116 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T18:20:38.405 INFO:teuthology.orchestra.run.smithi111.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 926 kB/s | 217 kB 00:00 2024-03-25T18:20:38.539 INFO:teuthology.orchestra.run.smithi111.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.2 MB/s | 300 kB 00:00 2024-03-25T18:20:38.589 INFO:teuthology.orchestra.run.smithi111.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-25T18:20:38.723 INFO:teuthology.orchestra.run.smithi111.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-25T18:20:39.465 INFO:teuthology.orchestra.run.smithi111.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 126 kB/s | 163 kB 00:01 2024-03-25T18:21:59.414 INFO:teuthology.orchestra.run.smithi111.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 596 kB/s | 47 MB 01:21 2024-03-25T18:21:59.416 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:21:59.416 INFO:teuthology.orchestra.run.smithi111.stdout:Total 607 kB/s | 48 MB 01:21 2024-03-25T18:21:59.427 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T18:21:59.447 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T18:21:59.447 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T18:21:59.825 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T18:21:59.825 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T18:22:00.402 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T18:22:00.472 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:22:00.511 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:22:00.568 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:22:00.619 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:22:00.650 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:22:05.413 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:22:05.482 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:22:07.723 INFO:teuthology.orchestra.run.smithi044.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 403 kB/s | 47 MB 02:00 2024-03-25T18:22:07.726 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:07.726 INFO:teuthology.orchestra.run.smithi044.stdout:Total 409 kB/s | 48 MB 02:00 2024-03-25T18:22:07.737 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T18:22:07.758 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T18:22:07.758 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T18:22:07.871 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:22:07.871 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:22:07.871 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:22:07.872 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:22:07.872 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:22:07.872 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:22:08.151 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T18:22:08.151 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T18:22:08.237 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:22:08.238 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:22:08.238 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T18:22:08.238 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:22:08.238 INFO:teuthology.orchestra.run.smithi111.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:22:08.238 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:22:08.238 INFO:teuthology.orchestra.run.smithi111.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:22:08.238 INFO:teuthology.orchestra.run.smithi111.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:22:08.238 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:22:08.238 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:22:08.238 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:22:08.561 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph 2024-03-25T18:22:08.758 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T18:22:08.841 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:22:08.876 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:22:08.908 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:22:08.958 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:22:08.984 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:22:09.125 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:49 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:22:09.290 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:22:09.292 INFO:teuthology.orchestra.run.smithi111.stdout:========================================================================================= 2024-03-25T18:22:09.292 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-25T18:22:09.292 INFO:teuthology.orchestra.run.smithi111.stdout:========================================================================================= 2024-03-25T18:22:09.292 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T18:22:09.293 INFO:teuthology.orchestra.run.smithi111.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:22:09.293 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-25T18:22:09.293 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:22:09.293 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:22:09.293 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:22:09.293 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:22:09.293 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:22:09.293 INFO:teuthology.orchestra.run.smithi111.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:22:09.293 INFO:teuthology.orchestra.run.smithi111.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:22:09.293 INFO:teuthology.orchestra.run.smithi111.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:22:09.293 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:22:09.293 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:22:09.293 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:22:09.294 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:22:09.294 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:22:09.294 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:22:09.294 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:22:09.294 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:22:09.294 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:22:09.294 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:22:09.294 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:22:09.294 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:22:09.294 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:22:09.295 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:22:09.295 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:22:09.295 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:22:09.295 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:22:09.295 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:22:09.295 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:22:09.295 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:22:09.295 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:22:09.295 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:22:09.295 INFO:teuthology.orchestra.run.smithi111.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:22:09.295 INFO:teuthology.orchestra.run.smithi111.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:22:09.295 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:22:09.295 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:22:09.296 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:22:09.296 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:22:09.296 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T18:22:09.296 INFO:teuthology.orchestra.run.smithi111.stdout:========================================================================================= 2024-03-25T18:22:09.296 INFO:teuthology.orchestra.run.smithi111.stdout:Install 36 Packages 2024-03-25T18:22:09.296 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:22:09.296 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 30 M 2024-03-25T18:22:09.297 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 107 M 2024-03-25T18:22:09.297 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T18:22:10.000 INFO:teuthology.orchestra.run.smithi111.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2024-03-25T18:22:13.679 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:22:13.752 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:22:14.009 INFO:teuthology.orchestra.run.smithi111.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 342 kB/s | 1.5 MB 00:04 2024-03-25T18:22:15.925 INFO:teuthology.orchestra.run.smithi111.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 341 kB/s | 2.1 MB 00:06 2024-03-25T18:22:16.051 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:22:16.051 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:22:16.051 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:22:16.051 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:22:16.051 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:22:16.051 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:22:16.389 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:22:16.389 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:22:16.389 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T18:22:16.389 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:22:16.389 INFO:teuthology.orchestra.run.smithi044.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:22:16.389 INFO:teuthology.orchestra.run.smithi044.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:22:16.389 INFO:teuthology.orchestra.run.smithi044.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:22:16.389 INFO:teuthology.orchestra.run.smithi044.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:22:16.389 INFO:teuthology.orchestra.run.smithi044.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:22:16.389 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:22:16.389 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:22:16.697 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph 2024-03-25T18:22:17.142 INFO:teuthology.orchestra.run.smithi111.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 203 kB/s | 247 kB 00:01 2024-03-25T18:22:17.267 INFO:teuthology.orchestra.run.smithi111.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T18:22:17.269 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:03:26 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:22:17.309 INFO:teuthology.orchestra.run.smithi111.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T18:22:17.334 INFO:teuthology.orchestra.run.smithi111.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-25T18:22:17.368 INFO:teuthology.orchestra.run.smithi111.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T18:22:17.393 INFO:teuthology.orchestra.run.smithi111.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-25T18:22:17.435 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:22:17.438 INFO:teuthology.orchestra.run.smithi044.stdout:========================================================================================= 2024-03-25T18:22:17.438 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-03-25T18:22:17.438 INFO:teuthology.orchestra.run.smithi044.stdout:========================================================================================= 2024-03-25T18:22:17.438 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T18:22:17.438 INFO:teuthology.orchestra.run.smithi044.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:22:17.438 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-03-25T18:22:17.438 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:22:17.438 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:22:17.438 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:22:17.438 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:22:17.438 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:22:17.438 INFO:teuthology.orchestra.run.smithi044.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:22:17.439 INFO:teuthology.orchestra.run.smithi044.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:22:17.439 INFO:teuthology.orchestra.run.smithi044.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:22:17.439 INFO:teuthology.orchestra.run.smithi044.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:22:17.439 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:22:17.439 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:22:17.439 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:22:17.439 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:22:17.439 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:22:17.439 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:22:17.439 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:22:17.439 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:22:17.439 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:22:17.439 INFO:teuthology.orchestra.run.smithi044.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:22:17.440 INFO:teuthology.orchestra.run.smithi044.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:22:17.440 INFO:teuthology.orchestra.run.smithi044.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:22:17.440 INFO:teuthology.orchestra.run.smithi044.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:22:17.440 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:22:17.440 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:22:17.440 INFO:teuthology.orchestra.run.smithi044.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:22:17.440 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:22:17.440 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:22:17.440 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:22:17.440 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:22:17.440 INFO:teuthology.orchestra.run.smithi044.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:22:17.440 INFO:teuthology.orchestra.run.smithi044.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:22:17.440 INFO:teuthology.orchestra.run.smithi044.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:22:17.440 INFO:teuthology.orchestra.run.smithi044.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:22:17.440 INFO:teuthology.orchestra.run.smithi044.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:22:17.441 INFO:teuthology.orchestra.run.smithi044.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:22:17.441 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:22:17.441 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T18:22:17.441 INFO:teuthology.orchestra.run.smithi044.stdout:========================================================================================= 2024-03-25T18:22:17.441 INFO:teuthology.orchestra.run.smithi044.stdout:Install 36 Packages 2024-03-25T18:22:17.441 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:22:17.441 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 30 M 2024-03-25T18:22:17.441 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 107 M 2024-03-25T18:22:17.442 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T18:22:17.452 INFO:teuthology.orchestra.run.smithi111.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-25T18:22:17.527 INFO:teuthology.orchestra.run.smithi111.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T18:22:17.561 INFO:teuthology.orchestra.run.smithi111.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T18:22:17.594 INFO:teuthology.orchestra.run.smithi111.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T18:22:17.636 INFO:teuthology.orchestra.run.smithi111.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T18:22:17.711 INFO:teuthology.orchestra.run.smithi111.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.6 MB/s | 125 kB 00:00 2024-03-25T18:22:17.787 INFO:teuthology.orchestra.run.smithi111.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 297 kB/s | 22 kB 00:00 2024-03-25T18:22:18.212 INFO:teuthology.orchestra.run.smithi111.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 355 kB/s | 151 kB 00:00 2024-03-25T18:22:18.384 INFO:teuthology.orchestra.run.smithi044.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 14 kB/s | 6.6 kB 00:00 2024-03-25T18:22:18.563 INFO:teuthology.orchestra.run.smithi111.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 124 kB/s | 43 kB 00:00 2024-03-25T18:22:18.988 INFO:teuthology.orchestra.run.smithi111.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 404 kB/s | 172 kB 00:00 2024-03-25T18:22:19.496 INFO:teuthology.orchestra.run.smithi111.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 705 kB/s | 358 kB 00:00 2024-03-25T18:22:19.680 INFO:teuthology.orchestra.run.smithi111.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 58 kB/s | 11 kB 00:00 2024-03-25T18:22:19.964 INFO:teuthology.orchestra.run.smithi111.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 62 kB/s | 18 kB 00:00 2024-03-25T18:22:20.222 INFO:teuthology.orchestra.run.smithi111.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 90 kB/s | 23 kB 00:00 2024-03-25T18:22:20.498 INFO:teuthology.orchestra.run.smithi111.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 71 kB/s | 19 kB 00:00 2024-03-25T18:22:20.756 INFO:teuthology.orchestra.run.smithi111.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 76 kB/s | 20 kB 00:00 2024-03-25T18:22:21.100 INFO:teuthology.orchestra.run.smithi111.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 135 kB/s | 46 kB 00:00 2024-03-25T18:22:21.459 INFO:teuthology.orchestra.run.smithi111.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 220 kB/s | 79 kB 00:00 2024-03-25T18:22:21.826 INFO:teuthology.orchestra.run.smithi111.stdout:(28/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 406 kB/s | 4.7 MB 00:11 2024-03-25T18:22:21.984 INFO:teuthology.orchestra.run.smithi111.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 518 kB/s | 272 kB 00:00 2024-03-25T18:22:22.093 INFO:teuthology.orchestra.run.smithi111.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 61 kB/s | 16 kB 00:00 2024-03-25T18:22:22.327 INFO:teuthology.orchestra.run.smithi111.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 263 kB/s | 90 kB 00:00 2024-03-25T18:22:22.360 INFO:teuthology.orchestra.run.smithi111.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 134 kB/s | 36 kB 00:00 2024-03-25T18:22:22.835 INFO:teuthology.orchestra.run.smithi111.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 451 kB/s | 230 kB 00:00 2024-03-25T18:22:22.952 INFO:teuthology.orchestra.run.smithi111.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 722 kB/s | 427 kB 00:00 2024-03-25T18:22:23.136 INFO:teuthology.orchestra.run.smithi111.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 66 kB/s | 20 kB 00:00 2024-03-25T18:22:23.450 INFO:teuthology.orchestra.run.smithi044.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 276 kB/s | 1.5 MB 00:05 2024-03-25T18:22:24.167 INFO:teuthology.orchestra.run.smithi044.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 348 kB/s | 2.1 MB 00:06 2024-03-25T18:22:24.792 INFO:teuthology.orchestra.run.smithi044.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 396 kB/s | 247 kB 00:00 2024-03-25T18:22:24.918 INFO:teuthology.orchestra.run.smithi044.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T18:22:24.960 INFO:teuthology.orchestra.run.smithi044.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T18:22:24.985 INFO:teuthology.orchestra.run.smithi044.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-25T18:22:25.010 INFO:teuthology.orchestra.run.smithi044.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2024-03-25T18:22:25.036 INFO:teuthology.orchestra.run.smithi044.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-25T18:22:25.078 INFO:teuthology.orchestra.run.smithi044.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-25T18:22:25.153 INFO:teuthology.orchestra.run.smithi044.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T18:22:25.186 INFO:teuthology.orchestra.run.smithi044.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T18:22:25.229 INFO:teuthology.orchestra.run.smithi044.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-25T18:22:25.270 INFO:teuthology.orchestra.run.smithi044.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T18:22:25.704 INFO:teuthology.orchestra.run.smithi044.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 288 kB/s | 125 kB 00:00 2024-03-25T18:22:25.829 INFO:teuthology.orchestra.run.smithi044.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 178 kB/s | 22 kB 00:00 2024-03-25T18:22:26.021 INFO:teuthology.orchestra.run.smithi044.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 788 kB/s | 151 kB 00:00 2024-03-25T18:22:26.071 INFO:teuthology.orchestra.run.smithi044.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 869 kB/s | 43 kB 00:00 2024-03-25T18:22:26.138 INFO:teuthology.orchestra.run.smithi044.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-03-25T18:22:26.205 INFO:teuthology.orchestra.run.smithi044.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 5.3 MB/s | 358 kB 00:00 2024-03-25T18:22:26.256 INFO:teuthology.orchestra.run.smithi044.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-03-25T18:22:26.306 INFO:teuthology.orchestra.run.smithi044.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 354 kB/s | 18 kB 00:00 2024-03-25T18:22:26.356 INFO:teuthology.orchestra.run.smithi044.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 465 kB/s | 23 kB 00:00 2024-03-25T18:22:26.406 INFO:teuthology.orchestra.run.smithi044.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 389 kB/s | 19 kB 00:00 2024-03-25T18:22:26.457 INFO:teuthology.orchestra.run.smithi044.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 395 kB/s | 20 kB 00:00 2024-03-25T18:22:26.507 INFO:teuthology.orchestra.run.smithi044.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 928 kB/s | 46 kB 00:00 2024-03-25T18:22:26.557 INFO:teuthology.orchestra.run.smithi044.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-03-25T18:22:26.608 INFO:teuthology.orchestra.run.smithi044.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-25T18:22:26.658 INFO:teuthology.orchestra.run.smithi044.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 329 kB/s | 16 kB 00:00 2024-03-25T18:22:26.708 INFO:teuthology.orchestra.run.smithi044.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-03-25T18:22:26.758 INFO:teuthology.orchestra.run.smithi044.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 715 kB/s | 36 kB 00:00 2024-03-25T18:22:26.809 INFO:teuthology.orchestra.run.smithi044.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-25T18:22:26.860 INFO:teuthology.orchestra.run.smithi044.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 8.4 MB/s | 427 kB 00:00 2024-03-25T18:22:26.918 INFO:teuthology.orchestra.run.smithi044.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 341 kB/s | 20 kB 00:00 2024-03-25T18:22:30.260 INFO:teuthology.orchestra.run.smithi044.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 404 kB/s | 4.7 MB 00:11 2024-03-25T18:22:52.966 INFO:teuthology.orchestra.run.smithi111.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 441 kB/s | 17 MB 00:38 2024-03-25T18:22:52.969 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:52.969 INFO:teuthology.orchestra.run.smithi111.stdout:Total 696 kB/s | 30 MB 00:43 2024-03-25T18:22:52.993 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T18:22:53.034 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T18:22:53.035 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T18:22:53.430 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T18:22:53.430 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T18:22:54.773 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T18:22:54.842 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:22:54.876 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:22:54.912 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:22:54.982 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:22:55.027 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:22:55.060 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:22:55.150 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:22:55.238 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:22:55.279 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:22:55.350 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:22:55.405 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:22:55.443 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:22:55.492 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:22:55.547 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:22:55.730 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:22:55.789 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:22:55.819 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:22:55.854 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:22:55.884 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:22:55.960 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:22:55.994 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:22:56.028 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:22:56.069 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:22:56.103 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:22:56.138 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:22:56.226 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:22:56.268 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:22:56.340 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:22:56.476 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:22:56.603 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:22:56.760 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:22:56.782 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:22:56.807 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:22:56.808 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:22:56.808 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:22:56.808 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:22:56.808 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:22:56.808 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:22:58.233 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:22:58.259 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:22:58.259 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:22:58.259 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:22:58.259 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:22:58.259 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:22:58.259 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:22:58.700 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:22:58.723 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:22:58.723 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:22:58.723 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:22:58.724 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:22:58.724 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:22:58.724 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:22:58.934 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:22:58.957 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:22:58.957 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:22:58.957 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:22:58.957 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:22:58.957 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:22:58.957 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:22:59.023 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:23:00.778 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:23:00.779 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:23:00.779 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:23:00.779 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:23:00.779 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:23:00.779 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:23:00.779 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:23:00.779 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:23:00.779 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:23:00.779 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:23:00.779 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:23:00.779 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:23:00.780 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:23:00.780 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:23:00.780 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:23:00.780 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:23:00.780 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:23:00.780 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:23:00.780 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:23:00.780 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:23:00.780 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:23:00.780 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:23:00.780 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:23:00.780 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:23:00.780 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:23:00.781 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:23:00.781 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:23:00.781 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:23:00.781 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:23:00.781 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:23:00.781 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:23:00.781 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:23:00.781 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:23:00.781 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:23:00.781 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:23:00.781 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:23:01.685 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:23:01.685 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:01.685 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T18:23:01.685 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:01.685 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:01.685 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:01.686 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:23:01.686 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:01.686 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:01.686 INFO:teuthology.orchestra.run.smithi111.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:23:01.686 INFO:teuthology.orchestra.run.smithi111.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:23:01.686 INFO:teuthology.orchestra.run.smithi111.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:23:01.686 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:23:01.686 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:23:01.686 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:23:01.686 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:23:01.686 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:23:01.686 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:23:01.687 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:23:01.687 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:23:01.687 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:23:01.687 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:23:01.687 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:23:01.687 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:23:01.687 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:23:01.687 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:23:01.687 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:23:01.687 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:23:01.687 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:23:01.687 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:23:01.687 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:23:01.687 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:23:01.688 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:23:01.688 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:23:01.688 INFO:teuthology.orchestra.run.smithi111.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:23:01.688 INFO:teuthology.orchestra.run.smithi111.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:23:01.688 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:23:01.688 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:23:01.688 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:23:01.688 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:01.688 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:23:02.015 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-base 2024-03-25T18:23:02.579 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:03:42 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:23:02.676 INFO:teuthology.orchestra.run.smithi111.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:23:02.728 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:23:02.729 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-25T18:23:02.729 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:23:02.799 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install cephadm 2024-03-25T18:23:03.357 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:03:43 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:23:03.505 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:23:03.507 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:23:03.507 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-25T18:23:03.507 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:23:03.507 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T18:23:03.507 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:23:03.507 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:03.507 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T18:23:03.507 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:23:03.507 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T18:23:03.507 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:03.507 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 747 k 2024-03-25T18:23:03.507 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 754 k 2024-03-25T18:23:03.507 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T18:23:05.258 INFO:teuthology.orchestra.run.smithi111.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 427 kB/s | 747 kB 00:01 2024-03-25T18:23:05.258 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:05.259 INFO:teuthology.orchestra.run.smithi111.stdout:Total 427 kB/s | 747 kB 00:01 2024-03-25T18:23:05.259 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T18:23:05.265 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T18:23:05.265 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T18:23:05.271 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T18:23:05.272 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T18:23:05.348 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T18:23:05.769 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:23:05.854 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:23:06.256 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:23:06.535 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:23:06.535 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:06.535 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T18:23:06.535 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:23:06.535 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:06.535 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:23:06.736 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:23:07.305 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:03:47 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:23:07.456 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:23:07.457 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:23:07.457 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T18:23:07.457 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:23:07.457 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T18:23:07.457 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:23:07.457 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:07.457 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T18:23:07.457 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:23:07.457 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T18:23:07.457 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:07.458 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 145 k 2024-03-25T18:23:07.458 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 436 k 2024-03-25T18:23:07.458 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T18:23:07.856 INFO:teuthology.orchestra.run.smithi044.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 387 kB/s | 17 MB 00:44 2024-03-25T18:23:07.860 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:07.860 INFO:teuthology.orchestra.run.smithi044.stdout:Total 602 kB/s | 30 MB 00:50 2024-03-25T18:23:07.883 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T18:23:07.926 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T18:23:07.926 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T18:23:08.005 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 266 kB/s | 145 kB 00:00 2024-03-25T18:23:08.006 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:08.006 INFO:teuthology.orchestra.run.smithi111.stdout:Total 265 kB/s | 145 kB 00:00 2024-03-25T18:23:08.006 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T18:23:08.014 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T18:23:08.014 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T18:23:08.067 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T18:23:08.067 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T18:23:08.202 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T18:23:08.281 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:23:08.305 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:23:08.306 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:23:08.306 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:23:08.306 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:08.345 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T18:23:08.345 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T18:23:09.328 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:23:09.329 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:09.329 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T18:23:09.329 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:09.329 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:09.329 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:23:09.547 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr 2024-03-25T18:23:09.686 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T18:23:09.742 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:23:09.778 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:23:09.811 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:23:09.884 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:23:09.927 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:23:09.959 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:23:10.055 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:23:10.118 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:03:50 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:23:10.140 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:23:10.188 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:23:10.214 INFO:teuthology.orchestra.run.smithi111.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:23:10.262 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:23:10.266 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:23:10.267 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-25T18:23:10.267 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:23:10.322 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:23:10.338 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:23:10.359 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:23:10.410 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:23:10.465 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:23:10.655 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:23:10.697 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:23:10.735 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:23:10.770 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:23:10.800 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:23:10.878 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:23:10.894 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:03:50 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:23:10.910 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:23:10.944 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:23:10.985 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:23:11.043 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:23:11.044 INFO:teuthology.orchestra.run.smithi111.stdout:====================================================================================== 2024-03-25T18:23:11.044 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-25T18:23:11.045 INFO:teuthology.orchestra.run.smithi111.stdout:====================================================================================== 2024-03-25T18:23:11.045 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T18:23:11.045 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:23:11.045 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-25T18:23:11.045 INFO:teuthology.orchestra.run.smithi111.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:23:11.045 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:23:11.045 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:23:11.045 INFO:teuthology.orchestra.run.smithi111.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:23:11.045 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:23:11.045 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:23:11.045 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:23:11.046 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:23:11.046 INFO:teuthology.orchestra.run.smithi111.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:23:11.046 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:23:11.046 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:23:11.046 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:11.046 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T18:23:11.046 INFO:teuthology.orchestra.run.smithi111.stdout:====================================================================================== 2024-03-25T18:23:11.046 INFO:teuthology.orchestra.run.smithi111.stdout:Install 12 Packages 2024-03-25T18:23:11.046 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:11.047 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 6.9 M 2024-03-25T18:23:11.047 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 26 M 2024-03-25T18:23:11.047 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T18:23:11.056 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:23:11.087 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:23:11.180 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:23:11.218 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:23:11.294 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:23:11.443 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:23:11.567 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:23:11.734 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:23:11.750 INFO:teuthology.orchestra.run.smithi111.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 59 kB/s | 31 kB 00:00 2024-03-25T18:23:11.756 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:23:11.776 INFO:teuthology.orchestra.run.smithi111.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 27 kB/s | 15 kB 00:00 2024-03-25T18:23:11.781 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:23:11.781 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:23:11.781 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:23:11.781 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:23:11.782 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:23:11.782 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:23:11.934 INFO:teuthology.orchestra.run.smithi111.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 238 kB 00:00 2024-03-25T18:23:11.968 INFO:teuthology.orchestra.run.smithi111.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-03-25T18:23:12.152 INFO:teuthology.orchestra.run.smithi111.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.9 MB/s | 548 kB 00:00 2024-03-25T18:23:12.193 INFO:teuthology.orchestra.run.smithi111.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 468 kB/s | 19 kB 00:00 2024-03-25T18:23:12.461 INFO:teuthology.orchestra.run.smithi111.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.6 MB/s | 863 kB 00:00 2024-03-25T18:23:12.519 INFO:teuthology.orchestra.run.smithi111.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 6.3 MB/s | 2.0 MB 00:00 2024-03-25T18:23:12.561 INFO:teuthology.orchestra.run.smithi111.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 744 kB/s | 31 kB 00:00 2024-03-25T18:23:12.586 INFO:teuthology.orchestra.run.smithi111.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.1 MB/s | 144 kB 00:00 2024-03-25T18:23:12.620 INFO:teuthology.orchestra.run.smithi111.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.1 MB/s | 188 kB 00:00 2024-03-25T18:23:13.307 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:23:13.334 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:23:13.334 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:23:13.334 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:23:13.334 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:23:13.334 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:23:13.335 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:23:13.800 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:23:13.822 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:23:13.822 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:23:13.822 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:23:13.822 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:23:13.822 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:23:13.822 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:23:14.034 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:23:14.056 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:23:14.056 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:23:14.057 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:23:14.057 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:23:14.057 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:23:14.057 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:23:14.104 INFO:teuthology.orchestra.run.smithi111.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 632 kB/s | 1.8 MB 00:02 2024-03-25T18:23:14.108 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:14.108 INFO:teuthology.orchestra.run.smithi111.stdout:Total 2.2 MB/s | 6.9 MB 00:03 2024-03-25T18:23:14.125 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:23:14.127 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T18:23:14.146 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T18:23:14.146 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T18:23:14.324 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T18:23:14.324 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T18:23:14.841 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T18:23:14.929 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:23:14.964 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:23:15.010 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:23:15.041 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:23:15.148 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:23:15.273 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:23:15.367 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:23:15.412 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:23:15.441 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:23:15.472 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:23:15.712 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:23:15.747 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:23:15.834 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:23:15.834 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:23:15.834 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:23:15.834 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:23:15.835 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:23:15.835 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:23:15.835 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:23:15.835 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:23:15.835 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:23:15.835 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:23:15.835 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:23:15.835 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:23:15.835 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:23:15.835 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:23:15.835 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:23:15.835 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:23:15.835 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:23:15.835 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:23:15.835 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:23:15.836 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:23:15.836 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:23:15.836 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:23:15.836 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:23:15.836 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:23:15.836 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:23:15.836 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:23:15.836 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:23:15.836 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:23:15.837 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:23:15.837 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:23:15.837 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:23:15.837 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:23:15.837 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:23:15.837 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:23:15.838 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:23:15.838 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:23:16.515 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:23:16.516 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:23:16.516 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:23:16.516 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:23:16.516 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:23:16.516 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:23:16.516 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:23:16.516 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:23:16.516 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:23:16.516 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:23:16.516 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:23:16.516 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:23:16.762 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:23:16.762 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:23:16.763 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T18:23:16.763 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:16.763 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:16.763 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:16.763 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:23:16.763 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:16.763 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:16.763 INFO:teuthology.orchestra.run.smithi044.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:23:16.763 INFO:teuthology.orchestra.run.smithi044.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:23:16.763 INFO:teuthology.orchestra.run.smithi044.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:23:16.763 INFO:teuthology.orchestra.run.smithi044.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:23:16.763 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:23:16.763 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:23:16.763 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:23:16.764 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:23:16.764 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:23:16.764 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:23:16.764 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:23:16.764 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:23:16.764 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:23:16.764 INFO:teuthology.orchestra.run.smithi044.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:23:16.764 INFO:teuthology.orchestra.run.smithi044.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:23:16.764 INFO:teuthology.orchestra.run.smithi044.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:23:16.764 INFO:teuthology.orchestra.run.smithi044.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:23:16.764 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:23:16.764 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:23:16.764 INFO:teuthology.orchestra.run.smithi044.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:23:16.764 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:23:16.765 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:23:16.765 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:23:16.765 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:23:16.765 INFO:teuthology.orchestra.run.smithi044.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:23:16.765 INFO:teuthology.orchestra.run.smithi044.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:23:16.765 INFO:teuthology.orchestra.run.smithi044.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:23:16.765 INFO:teuthology.orchestra.run.smithi044.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:23:16.765 INFO:teuthology.orchestra.run.smithi044.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:23:16.765 INFO:teuthology.orchestra.run.smithi044.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:23:16.765 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:23:16.765 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:23:16.868 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:23:16.868 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:16.868 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T18:23:16.868 INFO:teuthology.orchestra.run.smithi111.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:23:16.868 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:23:16.868 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:23:16.868 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:23:16.868 INFO:teuthology.orchestra.run.smithi111.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:23:16.869 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:23:16.869 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:23:16.869 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:23:16.869 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:23:16.869 INFO:teuthology.orchestra.run.smithi111.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:23:16.869 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:23:16.869 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:23:16.869 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:16.869 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:23:17.100 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-base 2024-03-25T18:23:17.142 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:23:17.674 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:04:26 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:23:17.700 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:03:57 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:23:17.773 INFO:teuthology.orchestra.run.smithi044.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:23:17.828 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:23:17.829 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-03-25T18:23:17.830 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:23:17.850 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:23:17.851 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================================ 2024-03-25T18:23:17.851 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-25T18:23:17.851 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================================ 2024-03-25T18:23:17.851 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T18:23:17.851 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:23:17.852 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-25T18:23:17.852 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:23:17.852 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:23:17.852 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:23:17.852 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:23:17.852 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:23:17.852 INFO:teuthology.orchestra.run.smithi111.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:23:17.852 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:23:17.852 INFO:teuthology.orchestra.run.smithi111.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:23:17.852 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:23:17.852 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:23:17.853 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:23:17.853 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:17.853 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T18:23:17.853 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================================ 2024-03-25T18:23:17.853 INFO:teuthology.orchestra.run.smithi111.stdout:Install 12 Packages 2024-03-25T18:23:17.853 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:17.854 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 37 M 2024-03-25T18:23:17.854 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 209 M 2024-03-25T18:23:17.854 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T18:23:17.902 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install cephadm 2024-03-25T18:23:18.060 INFO:teuthology.orchestra.run.smithi111.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 241 kB/s | 32 kB 00:00 2024-03-25T18:23:18.085 INFO:teuthology.orchestra.run.smithi111.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 691 kB/s | 17 kB 00:00 2024-03-25T18:23:18.111 INFO:teuthology.orchestra.run.smithi111.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.3 MB/s | 34 kB 00:00 2024-03-25T18:23:18.277 INFO:teuthology.orchestra.run.smithi111.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 29 MB/s | 4.7 MB 00:00 2024-03-25T18:23:18.420 INFO:teuthology.orchestra.run.smithi111.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-25T18:23:18.462 INFO:teuthology.orchestra.run.smithi111.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.3 MB/s | 217 kB 00:00 2024-03-25T18:23:18.474 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:04:27 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:23:18.549 INFO:teuthology.orchestra.run.smithi111.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.8 MB/s | 3.0 MB 00:00 2024-03-25T18:23:18.629 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:23:18.630 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:23:18.630 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-03-25T18:23:18.630 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:23:18.630 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T18:23:18.630 INFO:teuthology.orchestra.run.smithi044.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:23:18.630 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:23:18.631 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T18:23:18.631 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:23:18.631 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T18:23:18.631 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:23:18.631 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 747 k 2024-03-25T18:23:18.631 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 754 k 2024-03-25T18:23:18.631 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T18:23:18.729 INFO:teuthology.orchestra.run.smithi111.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.4 MB/s | 803 kB 00:00 2024-03-25T18:23:18.879 INFO:teuthology.orchestra.run.smithi111.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-25T18:23:18.904 INFO:teuthology.orchestra.run.smithi111.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-25T18:23:19.138 INFO:teuthology.orchestra.run.smithi111.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 951 kB/s | 245 kB 00:00 2024-03-25T18:23:20.241 INFO:teuthology.orchestra.run.smithi044.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 464 kB/s | 747 kB 00:01 2024-03-25T18:23:20.242 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:20.242 INFO:teuthology.orchestra.run.smithi044.stdout:Total 464 kB/s | 747 kB 00:01 2024-03-25T18:23:20.242 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T18:23:20.249 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T18:23:20.249 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T18:23:20.255 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T18:23:20.255 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T18:23:20.329 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T18:23:20.804 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:23:20.891 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:23:21.383 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:23:21.696 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:23:21.696 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:23:21.696 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T18:23:21.696 INFO:teuthology.orchestra.run.smithi044.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:23:21.696 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:23:21.696 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:23:21.929 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:23:22.501 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:04:31 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:23:22.658 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:23:22.658 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:23:22.659 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T18:23:22.659 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:23:22.659 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T18:23:22.659 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:23:22.659 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:23:22.659 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T18:23:22.659 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:23:22.659 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T18:23:22.659 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:23:22.659 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 145 k 2024-03-25T18:23:22.659 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 436 k 2024-03-25T18:23:22.659 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T18:23:23.264 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 241 kB/s | 145 kB 00:00 2024-03-25T18:23:23.264 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:23.265 INFO:teuthology.orchestra.run.smithi044.stdout:Total 240 kB/s | 145 kB 00:00 2024-03-25T18:23:23.265 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T18:23:23.273 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T18:23:23.273 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T18:23:23.327 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T18:23:23.327 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T18:23:23.485 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T18:23:23.557 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:23:23.580 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:23:23.580 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:23:23.580 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:23:23.580 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:23:24.507 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:23:24.507 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:23:24.507 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T18:23:24.507 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:24.507 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:23:24.508 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:23:24.759 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-mgr 2024-03-25T18:23:25.331 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:04:34 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:23:25.433 INFO:teuthology.orchestra.run.smithi044.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:23:25.488 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:23:25.489 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-03-25T18:23:25.489 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:23:25.562 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:23:26.134 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:04:35 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:23:26.290 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:23:26.291 INFO:teuthology.orchestra.run.smithi044.stdout:====================================================================================== 2024-03-25T18:23:26.291 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-03-25T18:23:26.291 INFO:teuthology.orchestra.run.smithi044.stdout:====================================================================================== 2024-03-25T18:23:26.291 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T18:23:26.291 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:23:26.291 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-03-25T18:23:26.291 INFO:teuthology.orchestra.run.smithi044.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:23:26.292 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:23:26.292 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:23:26.292 INFO:teuthology.orchestra.run.smithi044.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:23:26.292 INFO:teuthology.orchestra.run.smithi044.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:23:26.292 INFO:teuthology.orchestra.run.smithi044.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:23:26.292 INFO:teuthology.orchestra.run.smithi044.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:23:26.292 INFO:teuthology.orchestra.run.smithi044.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:23:26.292 INFO:teuthology.orchestra.run.smithi044.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:23:26.292 INFO:teuthology.orchestra.run.smithi044.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:23:26.292 INFO:teuthology.orchestra.run.smithi044.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:23:26.292 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:23:26.292 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T18:23:26.292 INFO:teuthology.orchestra.run.smithi044.stdout:====================================================================================== 2024-03-25T18:23:26.293 INFO:teuthology.orchestra.run.smithi044.stdout:Install 12 Packages 2024-03-25T18:23:26.293 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:23:26.293 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 6.9 M 2024-03-25T18:23:26.293 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 26 M 2024-03-25T18:23:26.293 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T18:23:27.018 INFO:teuthology.orchestra.run.smithi044.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 77 kB/s | 31 kB 00:00 2024-03-25T18:23:27.101 INFO:teuthology.orchestra.run.smithi044.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 31 kB/s | 15 kB 00:00 2024-03-25T18:23:27.293 INFO:teuthology.orchestra.run.smithi044.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 238 kB 00:00 2024-03-25T18:23:27.360 INFO:teuthology.orchestra.run.smithi044.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.9 MB/s | 1.0 MB 00:00 2024-03-25T18:23:27.685 INFO:teuthology.orchestra.run.smithi044.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.2 MB/s | 863 kB 00:00 2024-03-25T18:23:27.860 INFO:teuthology.orchestra.run.smithi044.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-03-25T18:23:27.910 INFO:teuthology.orchestra.run.smithi044.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 86 kB/s | 19 kB 00:00 2024-03-25T18:23:28.053 INFO:teuthology.orchestra.run.smithi044.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 11 MB/s | 2.0 MB 00:00 2024-03-25T18:23:28.078 INFO:teuthology.orchestra.run.smithi044.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 865 kB/s | 144 kB 00:00 2024-03-25T18:23:28.120 INFO:teuthology.orchestra.run.smithi044.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 461 kB/s | 31 kB 00:00 2024-03-25T18:23:28.187 INFO:teuthology.orchestra.run.smithi044.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.7 MB/s | 188 kB 00:00 2024-03-25T18:23:28.987 INFO:teuthology.orchestra.run.smithi044.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 767 kB/s | 1.8 MB 00:02 2024-03-25T18:23:28.992 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:28.992 INFO:teuthology.orchestra.run.smithi044.stdout:Total 2.5 MB/s | 6.9 MB 00:02 2024-03-25T18:23:29.012 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T18:23:29.032 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T18:23:29.032 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T18:23:29.214 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T18:23:29.215 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T18:23:29.875 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T18:23:29.962 INFO:teuthology.orchestra.run.smithi111.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 629 kB/s | 7.4 MB 00:12 2024-03-25T18:23:29.963 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:23:29.964 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:29.965 INFO:teuthology.orchestra.run.smithi111.stdout:Total 3.1 MB/s | 37 MB 00:12 2024-03-25T18:23:30.007 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:23:30.009 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T18:23:30.034 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T18:23:30.035 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T18:23:30.054 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:23:30.092 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:23:30.192 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:23:30.323 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:23:30.392 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T18:23:30.392 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T18:23:30.415 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:23:30.463 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:23:30.491 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:23:30.523 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:23:30.780 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:23:30.815 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:23:31.117 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T18:23:31.201 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:23:31.241 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:23:31.335 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:23:31.377 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:23:31.554 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:23:31.585 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:23:31.667 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:23:31.758 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:23:31.758 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:23:31.758 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:23:31.758 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:23:31.758 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:23:31.758 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:23:31.758 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:23:31.758 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:23:31.759 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:23:31.759 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:23:31.759 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:23:31.759 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:23:32.105 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:23:32.105 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:23:32.106 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T18:23:32.106 INFO:teuthology.orchestra.run.smithi044.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:23:32.106 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:23:32.106 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:23:32.106 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:23:32.106 INFO:teuthology.orchestra.run.smithi044.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:23:32.106 INFO:teuthology.orchestra.run.smithi044.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:23:32.106 INFO:teuthology.orchestra.run.smithi044.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:23:32.106 INFO:teuthology.orchestra.run.smithi044.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:23:32.106 INFO:teuthology.orchestra.run.smithi044.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:23:32.106 INFO:teuthology.orchestra.run.smithi044.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:23:32.106 INFO:teuthology.orchestra.run.smithi044.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:23:32.106 INFO:teuthology.orchestra.run.smithi044.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:23:32.107 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:23:32.107 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:23:32.307 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:23:32.351 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:23:32.430 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:23:32.922 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:04:41 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:23:33.079 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:23:33.080 INFO:teuthology.orchestra.run.smithi044.stdout:============================================================================================ 2024-03-25T18:23:33.081 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-03-25T18:23:33.081 INFO:teuthology.orchestra.run.smithi044.stdout:============================================================================================ 2024-03-25T18:23:33.081 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T18:23:33.081 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:23:33.081 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-03-25T18:23:33.081 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:23:33.081 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:23:33.081 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:23:33.081 INFO:teuthology.orchestra.run.smithi044.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:23:33.081 INFO:teuthology.orchestra.run.smithi044.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:23:33.081 INFO:teuthology.orchestra.run.smithi044.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:23:33.082 INFO:teuthology.orchestra.run.smithi044.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:23:33.082 INFO:teuthology.orchestra.run.smithi044.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:23:33.082 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:23:33.082 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:23:33.082 INFO:teuthology.orchestra.run.smithi044.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:23:33.082 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:23:33.082 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T18:23:33.082 INFO:teuthology.orchestra.run.smithi044.stdout:============================================================================================ 2024-03-25T18:23:33.082 INFO:teuthology.orchestra.run.smithi044.stdout:Install 12 Packages 2024-03-25T18:23:33.082 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:23:33.083 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 37 M 2024-03-25T18:23:33.083 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 209 M 2024-03-25T18:23:33.083 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T18:23:33.552 INFO:teuthology.orchestra.run.smithi044.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 239 kB/s | 32 kB 00:00 2024-03-25T18:23:33.585 INFO:teuthology.orchestra.run.smithi044.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 523 kB/s | 17 kB 00:00 2024-03-25T18:23:33.656 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:23:33.660 INFO:teuthology.orchestra.run.smithi044.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-25T18:23:33.686 INFO:teuthology.orchestra.run.smithi044.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 345 kB/s | 34 kB 00:00 2024-03-25T18:23:33.911 INFO:teuthology.orchestra.run.smithi044.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 19 MB/s | 4.7 MB 00:00 2024-03-25T18:23:33.978 INFO:teuthology.orchestra.run.smithi044.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 4.8 MB 00:00 2024-03-25T18:23:34.003 INFO:teuthology.orchestra.run.smithi044.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-25T18:23:34.370 INFO:teuthology.orchestra.run.smithi044.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-03-25T18:23:34.571 INFO:teuthology.orchestra.run.smithi044.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 977 kB/s | 195 kB 00:00 2024-03-25T18:23:34.604 INFO:teuthology.orchestra.run.smithi044.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 803 kB 00:00 2024-03-25T18:23:34.663 INFO:teuthology.orchestra.run.smithi044.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.6 MB/s | 245 kB 00:00 2024-03-25T18:23:34.697 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:23:34.734 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:23:37.304 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:23:37.305 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:23:37.305 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:23:37.305 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:23:37.305 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:23:37.305 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:23:37.305 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:23:37.305 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:23:37.305 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:23:37.305 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:23:37.306 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:23:37.306 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:23:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:23:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:37.811 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T18:23:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:23:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:23:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:23:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:23:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:23:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:23:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:23:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:23:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:23:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:23:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:23:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:23:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:37.812 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:23:38.189 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:23:38.755 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:04:18 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:23:38.904 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:23:38.905 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================== 2024-03-25T18:23:38.905 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-25T18:23:38.905 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================== 2024-03-25T18:23:38.906 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T18:23:38.906 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:23:38.906 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-25T18:23:38.906 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:23:38.906 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:23:38.906 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:23:38.906 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:23:38.906 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:23:38.906 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:23:38.906 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:23:38.907 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:23:38.907 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:23:38.907 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:23:38.907 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:23:38.907 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:23:38.907 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:38.907 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T18:23:38.907 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================== 2024-03-25T18:23:38.907 INFO:teuthology.orchestra.run.smithi111.stdout:Install 13 Packages 2024-03-25T18:23:38.907 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:38.908 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 2.0 M 2024-03-25T18:23:38.908 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 26 M 2024-03-25T18:23:38.908 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T18:23:39.351 INFO:teuthology.orchestra.run.smithi111.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 260 kB/s | 24 kB 00:00 2024-03-25T18:23:39.401 INFO:teuthology.orchestra.run.smithi111.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 126 kB/s | 18 kB 00:00 2024-03-25T18:23:39.451 INFO:teuthology.orchestra.run.smithi111.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2024-03-25T18:23:39.476 INFO:teuthology.orchestra.run.smithi111.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T18:23:39.502 INFO:teuthology.orchestra.run.smithi111.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.2 MB/s | 211 kB 00:00 2024-03-25T18:23:39.527 INFO:teuthology.orchestra.run.smithi111.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 851 kB/s | 43 kB 00:00 2024-03-25T18:23:39.769 INFO:teuthology.orchestra.run.smithi111.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 62 kB/s | 15 kB 00:00 2024-03-25T18:23:39.794 INFO:teuthology.orchestra.run.smithi111.stdout:(8/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 97 kB/s | 52 kB 00:00 2024-03-25T18:23:39.820 INFO:teuthology.orchestra.run.smithi111.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 101 kB/s | 32 kB 00:00 2024-03-25T18:23:40.128 INFO:teuthology.orchestra.run.smithi111.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 192 kB/s | 59 kB 00:00 2024-03-25T18:23:40.153 INFO:teuthology.orchestra.run.smithi111.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 527 kB/s | 203 kB 00:00 2024-03-25T18:23:40.254 INFO:teuthology.orchestra.run.smithi111.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 716 kB/s | 90 kB 00:00 2024-03-25T18:23:40.304 INFO:teuthology.orchestra.run.smithi111.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.1 MB/s | 1.0 MB 00:00 2024-03-25T18:23:40.306 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:40.306 INFO:teuthology.orchestra.run.smithi111.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-03-25T18:23:40.306 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T18:23:40.319 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T18:23:40.319 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T18:23:40.407 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T18:23:40.408 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T18:23:40.916 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T18:23:41.074 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:23:41.121 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:23:41.185 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:23:41.222 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:23:41.263 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:23:41.357 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:23:41.457 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:23:41.515 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:23:42.230 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:23:42.287 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:23:42.321 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:23:42.369 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:23:42.391 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:23:43.162 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:23:43.162 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:23:43.162 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:23:43.162 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:23:43.162 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:23:43.163 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:23:43.163 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:23:43.163 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:23:43.163 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:23:43.163 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:23:43.163 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:23:43.163 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:23:43.163 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:23:43.515 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:23:43.515 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:43.515 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T18:23:43.515 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:23:43.515 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:23:43.515 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:23:43.515 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:23:43.515 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:23:43.515 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:23:43.515 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:23:43.515 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:23:43.516 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:23:43.516 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:23:43.516 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:23:43.516 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:23:43.516 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:23:43.516 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:43.516 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:23:43.767 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:23:44.332 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:04:24 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:23:44.482 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:23:44.483 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================== 2024-03-25T18:23:44.483 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-25T18:23:44.483 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================== 2024-03-25T18:23:44.483 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T18:23:44.483 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:23:44.483 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-25T18:23:44.483 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:23:44.484 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:23:44.484 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:23:44.484 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:23:44.484 INFO:teuthology.orchestra.run.smithi111.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:23:44.484 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:44.484 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T18:23:44.484 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================== 2024-03-25T18:23:44.484 INFO:teuthology.orchestra.run.smithi111.stdout:Install 6 Packages 2024-03-25T18:23:44.484 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:44.485 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 6.8 M 2024-03-25T18:23:44.485 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 33 M 2024-03-25T18:23:44.485 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T18:23:44.980 INFO:teuthology.orchestra.run.smithi111.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 2.3 MB/s | 248 kB 00:00 2024-03-25T18:23:45.181 INFO:teuthology.orchestra.run.smithi111.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-25T18:23:45.522 INFO:teuthology.orchestra.run.smithi111.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 169 kB/s | 58 kB 00:00 2024-03-25T18:23:45.581 INFO:teuthology.orchestra.run.smithi111.stdout:(4/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 195 kB/s | 138 kB 00:00 2024-03-25T18:23:45.606 INFO:teuthology.orchestra.run.smithi111.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 911 kB/s | 548 kB 00:00 2024-03-25T18:23:45.873 INFO:teuthology.orchestra.run.smithi111.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 145 kB/s | 51 kB 00:00 2024-03-25T18:23:45.875 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:45.875 INFO:teuthology.orchestra.run.smithi111.stdout:Total 4.9 MB/s | 6.8 MB 00:01 2024-03-25T18:23:45.888 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T18:23:45.898 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T18:23:45.898 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T18:23:45.990 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T18:23:45.990 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T18:23:46.282 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T18:23:46.437 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:23:46.483 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:23:46.991 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:23:47.078 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:23:47.147 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:23:47.171 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:23:47.955 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:23:47.956 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:23:47.956 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:23:47.956 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:23:47.956 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:23:47.956 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:23:48.253 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:23:48.253 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:48.253 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T18:23:48.253 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:23:48.253 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:23:48.253 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:23:48.253 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:23:48.254 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:23:48.254 INFO:teuthology.orchestra.run.smithi111.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:23:48.254 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:48.254 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:23:48.478 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-fuse 2024-03-25T18:23:49.043 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:04:29 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:23:49.193 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:23:49.195 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:23:49.195 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-25T18:23:49.195 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:23:49.195 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T18:23:49.195 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:23:49.195 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-25T18:23:49.195 INFO:teuthology.orchestra.run.smithi111.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:23:49.195 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:49.195 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T18:23:49.195 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:23:49.195 INFO:teuthology.orchestra.run.smithi111.stdout:Install 2 Packages 2024-03-25T18:23:49.195 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:49.196 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 964 k 2024-03-25T18:23:49.196 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 2.8 M 2024-03-25T18:23:49.196 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T18:23:49.358 INFO:teuthology.orchestra.run.smithi111.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 485 kB/s | 79 kB 00:00 2024-03-25T18:23:52.458 INFO:teuthology.orchestra.run.smithi111.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 271 kB/s | 885 kB 00:03 2024-03-25T18:23:52.459 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:52.460 INFO:teuthology.orchestra.run.smithi111.stdout:Total 295 kB/s | 964 kB 00:03 2024-03-25T18:23:52.460 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T18:23:52.484 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T18:23:52.484 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T18:23:52.559 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T18:23:52.560 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T18:23:52.745 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T18:23:52.828 INFO:teuthology.orchestra.run.smithi044.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 390 kB/s | 7.4 MB 00:19 2024-03-25T18:23:52.830 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:52.830 INFO:teuthology.orchestra.run.smithi044.stdout:Total 1.9 MB/s | 37 MB 00:19 2024-03-25T18:23:52.873 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T18:23:52.883 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:23:52.900 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T18:23:52.901 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T18:23:52.958 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:23:53.275 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T18:23:53.275 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T18:23:53.691 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:23:53.691 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:23:54.043 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:23:54.043 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:54.043 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T18:23:54.043 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:23:54.044 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:54.044 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:23:54.078 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T18:23:54.143 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:23:54.174 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:23:54.271 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:23:54.310 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:23:54.320 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-volume 2024-03-25T18:23:54.494 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:23:54.526 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:23:54.608 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:23:54.900 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:04:34 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:23:55.049 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:23:55.050 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:23:55.050 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-25T18:23:55.050 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:23:55.050 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T18:23:55.050 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:23:55.050 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-25T18:23:55.050 INFO:teuthology.orchestra.run.smithi111.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:23:55.050 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:55.050 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T18:23:55.050 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:23:55.050 INFO:teuthology.orchestra.run.smithi111.stdout:Install 2 Packages 2024-03-25T18:23:55.050 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:55.051 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 332 k 2024-03-25T18:23:55.051 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 1.5 M 2024-03-25T18:23:55.051 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T18:23:55.154 INFO:teuthology.orchestra.run.smithi111.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 669 kB/s | 69 kB 00:00 2024-03-25T18:23:55.288 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:23:55.400 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:23:56.171 INFO:teuthology.orchestra.run.smithi111.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 236 kB/s | 264 kB 00:01 2024-03-25T18:23:56.172 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:56.172 INFO:teuthology.orchestra.run.smithi111.stdout:Total 296 kB/s | 332 kB 00:01 2024-03-25T18:23:56.173 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T18:23:56.182 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T18:23:56.182 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T18:23:56.220 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T18:23:56.220 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T18:23:56.367 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T18:23:56.545 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:23:56.598 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:23:56.615 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:23:56.615 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:23:56.615 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:23:56.615 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:56.694 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:23:57.339 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:23:57.704 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:23:57.704 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:57.704 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T18:23:57.704 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:23:57.704 INFO:teuthology.orchestra.run.smithi111.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:23:57.704 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:57.704 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:23:57.768 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:23:57.800 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:23:58.035 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librados-devel 2024-03-25T18:23:58.605 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:04:38 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:23:58.755 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:23:58.756 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:23:58.756 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T18:23:58.756 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:23:58.756 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T18:23:58.756 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:23:58.756 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:58.757 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T18:23:58.757 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:23:58.757 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T18:23:58.757 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:23:58.757 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 127 k 2024-03-25T18:23:58.757 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 456 k 2024-03-25T18:23:58.758 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T18:23:59.651 INFO:teuthology.orchestra.run.smithi111.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 142 kB/s | 127 kB 00:00 2024-03-25T18:23:59.652 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:59.652 INFO:teuthology.orchestra.run.smithi111.stdout:Total 141 kB/s | 127 kB 00:00 2024-03-25T18:23:59.653 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T18:23:59.658 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T18:23:59.658 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T18:23:59.715 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T18:23:59.715 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T18:23:59.863 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T18:23:59.966 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:00.395 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:24:00.395 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:24:00.395 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:24:00.395 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:24:00.395 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:24:00.396 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:24:00.396 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:24:00.396 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:24:00.396 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:24:00.396 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:24:00.396 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:24:00.396 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:24:00.470 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:00.763 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:00.763 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:24:00.763 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T18:24:00.763 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:24:00.763 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:24:00.763 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:24:00.876 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:24:00.876 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:00.877 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T18:24:00.877 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:24:00.877 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:24:00.877 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:24:00.877 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:24:00.877 INFO:teuthology.orchestra.run.smithi044.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:24:00.877 INFO:teuthology.orchestra.run.smithi044.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:24:00.877 INFO:teuthology.orchestra.run.smithi044.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:24:00.877 INFO:teuthology.orchestra.run.smithi044.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:24:00.877 INFO:teuthology.orchestra.run.smithi044.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:24:00.877 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:24:00.877 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:24:00.877 INFO:teuthology.orchestra.run.smithi044.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:24:00.877 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:00.877 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:24:01.032 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install libcephfs2 2024-03-25T18:24:01.266 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:24:01.597 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:04:41 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:24:01.694 INFO:teuthology.orchestra.run.smithi111.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:24:01.746 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:24:01.748 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-25T18:24:01.748 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:24:01.829 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install libcephfs-devel 2024-03-25T18:24:01.839 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:05:10 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:24:01.998 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:24:01.999 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================== 2024-03-25T18:24:02.000 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-03-25T18:24:02.000 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================== 2024-03-25T18:24:02.000 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T18:24:02.000 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:24:02.000 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-03-25T18:24:02.000 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:24:02.000 INFO:teuthology.orchestra.run.smithi044.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:24:02.000 INFO:teuthology.orchestra.run.smithi044.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:24:02.000 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:24:02.000 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:24:02.000 INFO:teuthology.orchestra.run.smithi044.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:24:02.001 INFO:teuthology.orchestra.run.smithi044.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:24:02.001 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:24:02.001 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:24:02.001 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:24:02.001 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:24:02.001 INFO:teuthology.orchestra.run.smithi044.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:24:02.001 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:02.001 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T18:24:02.001 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================== 2024-03-25T18:24:02.001 INFO:teuthology.orchestra.run.smithi044.stdout:Install 13 Packages 2024-03-25T18:24:02.001 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:02.002 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 2.0 M 2024-03-25T18:24:02.002 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 26 M 2024-03-25T18:24:02.002 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T18:24:02.192 INFO:teuthology.orchestra.run.smithi044.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 152 kB/s | 18 kB 00:00 2024-03-25T18:24:02.242 INFO:teuthology.orchestra.run.smithi044.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 144 kB/s | 24 kB 00:00 2024-03-25T18:24:02.292 INFO:teuthology.orchestra.run.smithi044.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2024-03-25T18:24:02.317 INFO:teuthology.orchestra.run.smithi044.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T18:24:02.343 INFO:teuthology.orchestra.run.smithi044.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T18:24:02.368 INFO:teuthology.orchestra.run.smithi044.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 870 kB/s | 43 kB 00:00 2024-03-25T18:24:02.390 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:04:42 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:24:02.539 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:24:02.540 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:24:02.540 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T18:24:02.540 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:24:02.540 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T18:24:02.540 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:24:02.541 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:24:02.541 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T18:24:02.541 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:24:02.541 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T18:24:02.541 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:24:02.541 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 31 k 2024-03-25T18:24:02.541 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 140 k 2024-03-25T18:24:02.541 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T18:24:02.585 INFO:teuthology.orchestra.run.smithi044.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 69 kB/s | 15 kB 00:00 2024-03-25T18:24:02.660 INFO:teuthology.orchestra.run.smithi044.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 101 kB/s | 32 kB 00:00 2024-03-25T18:24:02.711 INFO:teuthology.orchestra.run.smithi044.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 82 kB/s | 52 kB 00:00 2024-03-25T18:24:02.827 INFO:teuthology.orchestra.run.smithi044.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 836 kB/s | 203 kB 00:00 2024-03-25T18:24:02.927 INFO:teuthology.orchestra.run.smithi044.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 906 kB/s | 90 kB 00:00 2024-03-25T18:24:02.990 INFO:teuthology.orchestra.run.smithi111.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 69 kB/s | 31 kB 00:00 2024-03-25T18:24:02.990 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:24:02.991 INFO:teuthology.orchestra.run.smithi111.stdout:Total 69 kB/s | 31 kB 00:00 2024-03-25T18:24:02.991 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T18:24:02.994 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T18:24:02.995 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T18:24:03.003 INFO:teuthology.orchestra.run.smithi044.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.0 MB/s | 1.0 MB 00:00 2024-03-25T18:24:03.019 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T18:24:03.020 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T18:24:03.028 INFO:teuthology.orchestra.run.smithi044.stdout:(13/13): python3-rsa-4.9-2.el9.noarch.rpm 186 kB/s | 59 kB 00:00 2024-03-25T18:24:03.033 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:24:03.033 INFO:teuthology.orchestra.run.smithi044.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2024-03-25T18:24:03.033 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T18:24:03.047 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T18:24:03.047 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T18:24:03.141 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T18:24:03.142 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T18:24:03.165 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T18:24:03.271 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:03.632 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:03.654 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T18:24:03.832 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:24:03.890 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:24:03.902 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:03.902 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:24:03.902 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T18:24:03.902 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:24:03.902 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:24:03.902 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:24:03.945 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:24:03.981 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:24:04.023 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:24:04.064 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librados2 2024-03-25T18:24:04.119 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:24:04.228 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:24:04.282 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:24:04.640 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:04:44 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:24:04.744 INFO:teuthology.orchestra.run.smithi111.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:24:04.800 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:24:04.801 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-25T18:24:04.801 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:24:04.856 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librbd1 2024-03-25T18:24:05.037 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:24:05.087 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:24:05.121 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:24:05.169 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:24:05.198 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:24:05.434 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:04:45 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:24:05.535 INFO:teuthology.orchestra.run.smithi111.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:24:05.589 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:24:05.590 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-25T18:24:05.590 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:24:05.642 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rados 2024-03-25T18:24:05.966 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:24:05.966 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:24:05.966 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:24:05.966 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:24:05.966 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:24:05.966 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:24:05.966 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:24:05.966 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:24:05.966 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:24:05.966 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:24:05.967 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:24:05.967 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:24:05.967 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:24:06.199 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:04:46 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:24:06.295 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:24:06.328 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:24:06.328 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:06.328 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T18:24:06.329 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:24:06.329 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:24:06.329 INFO:teuthology.orchestra.run.smithi044.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:24:06.329 INFO:teuthology.orchestra.run.smithi044.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:24:06.329 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:24:06.329 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:24:06.329 INFO:teuthology.orchestra.run.smithi044.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:24:06.329 INFO:teuthology.orchestra.run.smithi044.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:24:06.329 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:24:06.329 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:24:06.329 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:24:06.329 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:24:06.330 INFO:teuthology.orchestra.run.smithi044.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:24:06.330 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:06.330 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:24:06.346 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:24:06.347 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-25T18:24:06.347 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:24:06.404 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rgw 2024-03-25T18:24:06.562 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:24:06.962 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:04:46 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:24:07.057 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:24:07.108 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:24:07.109 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-25T18:24:07.110 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:24:07.135 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:05:16 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:24:07.159 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-cephfs 2024-03-25T18:24:07.292 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:24:07.293 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================== 2024-03-25T18:24:07.293 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-03-25T18:24:07.293 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================== 2024-03-25T18:24:07.293 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T18:24:07.293 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:24:07.293 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-03-25T18:24:07.293 INFO:teuthology.orchestra.run.smithi044.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:24:07.293 INFO:teuthology.orchestra.run.smithi044.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:24:07.293 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:24:07.293 INFO:teuthology.orchestra.run.smithi044.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:24:07.293 INFO:teuthology.orchestra.run.smithi044.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:24:07.293 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:07.294 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T18:24:07.294 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================== 2024-03-25T18:24:07.294 INFO:teuthology.orchestra.run.smithi044.stdout:Install 6 Packages 2024-03-25T18:24:07.294 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:07.294 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 6.8 M 2024-03-25T18:24:07.294 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 33 M 2024-03-25T18:24:07.295 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T18:24:07.716 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:04:47 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:24:07.738 INFO:teuthology.orchestra.run.smithi044.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-03-25T18:24:07.812 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:24:07.830 INFO:teuthology.orchestra.run.smithi044.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 5.8 MB 00:00 2024-03-25T18:24:07.864 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:24:07.865 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-25T18:24:07.865 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:24:07.920 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rbd 2024-03-25T18:24:07.989 INFO:teuthology.orchestra.run.smithi044.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 367 kB/s | 58 kB 00:00 2024-03-25T18:24:08.064 INFO:teuthology.orchestra.run.smithi044.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 681 kB/s | 51 kB 00:00 2024-03-25T18:24:08.089 INFO:teuthology.orchestra.run.smithi044.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2024-03-25T18:24:08.406 INFO:teuthology.orchestra.run.smithi044.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 166 kB/s | 138 kB 00:00 2024-03-25T18:24:08.407 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:24:08.408 INFO:teuthology.orchestra.run.smithi044.stdout:Total 6.1 MB/s | 6.8 MB 00:01 2024-03-25T18:24:08.421 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T18:24:08.432 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T18:24:08.432 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T18:24:08.473 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:04:48 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:24:08.525 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T18:24:08.525 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T18:24:08.569 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:24:08.620 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:24:08.621 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-25T18:24:08.621 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:24:08.666 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-fuse 2024-03-25T18:24:08.993 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T18:24:09.172 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:24:09.223 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:04:49 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:24:09.226 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:24:09.370 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:24:09.370 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:24:09.370 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-03-25T18:24:09.370 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:24:09.371 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T18:24:09.371 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:24:09.371 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:24:09.371 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T18:24:09.371 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:24:09.371 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T18:24:09.371 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:24:09.372 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 85 k 2024-03-25T18:24:09.372 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 231 k 2024-03-25T18:24:09.372 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T18:24:09.773 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:24:09.846 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:24:09.918 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:24:09.947 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:24:10.085 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 119 kB/s | 85 kB 00:00 2024-03-25T18:24:10.085 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:24:10.086 INFO:teuthology.orchestra.run.smithi111.stdout:Total 119 kB/s | 85 kB 00:00 2024-03-25T18:24:10.086 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T18:24:10.093 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T18:24:10.093 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T18:24:10.149 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T18:24:10.150 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T18:24:10.284 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T18:24:10.383 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:10.710 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:24:10.710 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:24:10.710 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:24:10.710 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:24:10.710 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:24:10.711 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:24:10.817 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:11.015 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:24:11.016 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:11.016 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T18:24:11.016 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:24:11.016 INFO:teuthology.orchestra.run.smithi044.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:24:11.016 INFO:teuthology.orchestra.run.smithi044.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:24:11.016 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:24:11.016 INFO:teuthology.orchestra.run.smithi044.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:24:11.016 INFO:teuthology.orchestra.run.smithi044.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:24:11.016 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:11.016 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:24:11.117 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:11.117 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:24:11.117 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T18:24:11.117 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:24:11.118 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:24:11.118 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:24:11.297 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-fuse 2024-03-25T18:24:11.369 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-mirror 2024-03-25T18:24:11.872 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:05:20 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:24:11.934 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:04:51 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:24:12.028 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:24:12.028 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:24:12.028 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-03-25T18:24:12.029 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:24:12.029 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T18:24:12.029 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:24:12.029 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-03-25T18:24:12.029 INFO:teuthology.orchestra.run.smithi044.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:24:12.029 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:12.029 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T18:24:12.029 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:24:12.029 INFO:teuthology.orchestra.run.smithi044.stdout:Install 2 Packages 2024-03-25T18:24:12.029 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:12.030 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 964 k 2024-03-25T18:24:12.030 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 2.8 M 2024-03-25T18:24:12.030 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T18:24:12.088 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:24:12.089 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:24:12.089 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T18:24:12.089 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:24:12.089 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T18:24:12.089 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:24:12.090 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:24:12.090 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T18:24:12.090 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:24:12.090 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T18:24:12.090 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:24:12.090 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 3.1 M 2024-03-25T18:24:12.090 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 13 M 2024-03-25T18:24:12.090 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T18:24:12.151 INFO:teuthology.orchestra.run.smithi044.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 649 kB/s | 79 kB 00:00 2024-03-25T18:24:14.534 INFO:teuthology.orchestra.run.smithi044.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 353 kB/s | 885 kB 00:02 2024-03-25T18:24:14.536 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:24:14.536 INFO:teuthology.orchestra.run.smithi044.stdout:Total 384 kB/s | 964 kB 00:02 2024-03-25T18:24:14.537 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T18:24:14.561 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T18:24:14.561 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T18:24:14.642 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T18:24:14.642 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T18:24:14.855 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T18:24:14.998 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:24:15.079 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:24:15.750 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:24:15.751 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:24:16.032 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:24:16.033 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:16.033 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T18:24:16.033 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:24:16.033 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:16.033 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:24:16.289 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-volume 2024-03-25T18:24:16.846 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 665 kB/s | 3.1 MB 00:04 2024-03-25T18:24:16.847 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:24:16.847 INFO:teuthology.orchestra.run.smithi111.stdout:Total 665 kB/s | 3.1 MB 00:04 2024-03-25T18:24:16.847 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T18:24:16.858 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T18:24:16.858 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T18:24:16.868 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:05:25 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:24:16.935 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T18:24:16.935 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T18:24:17.024 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:24:17.025 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:24:17.025 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-03-25T18:24:17.025 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:24:17.025 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T18:24:17.025 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:24:17.025 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-03-25T18:24:17.025 INFO:teuthology.orchestra.run.smithi044.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:24:17.025 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:17.025 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T18:24:17.025 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:24:17.025 INFO:teuthology.orchestra.run.smithi044.stdout:Install 2 Packages 2024-03-25T18:24:17.025 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:17.026 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 332 k 2024-03-25T18:24:17.026 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 1.5 M 2024-03-25T18:24:17.026 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T18:24:17.139 INFO:teuthology.orchestra.run.smithi044.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 608 kB/s | 69 kB 00:00 2024-03-25T18:24:17.375 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T18:24:17.430 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:17.454 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:17.454 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:24:17.454 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:24:17.454 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:24:17.454 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:24:17.455 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:24:18.306 INFO:teuthology.orchestra.run.smithi044.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 206 kB/s | 264 kB 00:01 2024-03-25T18:24:18.308 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:24:18.308 INFO:teuthology.orchestra.run.smithi044.stdout:Total 259 kB/s | 332 kB 00:01 2024-03-25T18:24:18.308 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T18:24:18.319 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T18:24:18.319 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T18:24:18.363 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T18:24:18.364 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T18:24:18.522 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T18:24:18.609 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:18.609 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:24:18.609 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T18:24:18.609 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:24:18.609 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:24:18.609 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:24:18.703 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:24:18.734 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:24:18.749 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:24:18.750 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:24:18.750 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:24:18.750 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:18.867 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-nbd 2024-03-25T18:24:19.448 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:04:59 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:24:19.456 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:24:19.595 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:24:19.596 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:24:19.596 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-03-25T18:24:19.596 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:24:19.596 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T18:24:19.596 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:24:19.596 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:24:19.596 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T18:24:19.596 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:24:19.596 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T18:24:19.596 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:24:19.596 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 171 k 2024-03-25T18:24:19.597 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 494 k 2024-03-25T18:24:19.597 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T18:24:19.791 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:24:19.791 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:19.791 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T18:24:19.791 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:24:19.791 INFO:teuthology.orchestra.run.smithi044.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:24:19.791 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:19.791 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:24:20.113 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install librados-devel 2024-03-25T18:24:20.499 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 190 kB/s | 171 kB 00:00 2024-03-25T18:24:20.500 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:24:20.500 INFO:teuthology.orchestra.run.smithi111.stdout:Total 190 kB/s | 171 kB 00:00 2024-03-25T18:24:20.500 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T18:24:20.510 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T18:24:20.510 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T18:24:20.570 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T18:24:20.570 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T18:24:20.695 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:05:29 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:24:20.718 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T18:24:20.822 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:20.851 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:24:20.852 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:24:20.852 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T18:24:20.852 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:24:20.852 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T18:24:20.852 INFO:teuthology.orchestra.run.smithi044.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:24:20.852 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:20.852 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T18:24:20.852 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:24:20.853 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T18:24:20.853 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:20.853 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 127 k 2024-03-25T18:24:20.853 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 456 k 2024-03-25T18:24:20.853 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T18:24:21.258 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:21.565 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:21.565 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:24:21.565 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T18:24:21.565 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:24:21.565 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:24:21.565 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:24:21.687 INFO:teuthology.orchestra.run.smithi044.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 152 kB/s | 127 kB 00:00 2024-03-25T18:24:21.687 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:24:21.688 INFO:teuthology.orchestra.run.smithi044.stdout:Total 152 kB/s | 127 kB 00:00 2024-03-25T18:24:21.688 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T18:24:21.694 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T18:24:21.694 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T18:24:21.758 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T18:24:21.758 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T18:24:21.825 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:24:21.915 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T18:24:22.022 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:22.389 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:05:02 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:24:22.426 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:22.540 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:24:22.541 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:24:22.541 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T18:24:22.541 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:24:22.541 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T18:24:22.541 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:24:22.541 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-25T18:24:22.541 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:24:22.541 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:24:22.541 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T18:24:22.541 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:24:22.541 INFO:teuthology.orchestra.run.smithi111.stdout:Install 2 Packages 2024-03-25T18:24:22.542 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:24:22.542 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 91 M 2024-03-25T18:24:22.542 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 498 M 2024-03-25T18:24:22.542 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T18:24:22.730 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:22.730 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:22.730 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T18:24:22.730 INFO:teuthology.orchestra.run.smithi044.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:24:22.730 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:22.730 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:24:22.978 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install libcephfs2 2024-03-25T18:24:23.554 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:05:32 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:24:23.655 INFO:teuthology.orchestra.run.smithi044.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:24:23.712 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:24:23.714 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-03-25T18:24:23.714 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:24:23.788 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install libcephfs-devel 2024-03-25T18:24:24.360 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:05:33 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:24:24.515 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:24:24.515 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:24:24.516 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T18:24:24.516 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:24:24.516 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T18:24:24.516 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:24:24.516 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:24.516 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T18:24:24.516 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:24:24.516 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T18:24:24.516 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:24.517 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 31 k 2024-03-25T18:24:24.517 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 140 k 2024-03-25T18:24:24.517 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T18:24:25.048 INFO:teuthology.orchestra.run.smithi044.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 59 kB/s | 31 kB 00:00 2024-03-25T18:24:25.049 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:24:25.049 INFO:teuthology.orchestra.run.smithi044.stdout:Total 58 kB/s | 31 kB 00:00 2024-03-25T18:24:25.049 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T18:24:25.053 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T18:24:25.053 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T18:24:25.078 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T18:24:25.079 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T18:24:25.213 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T18:24:25.367 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:25.742 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:26.014 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:26.014 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:26.015 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T18:24:26.015 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:24:26.015 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:26.015 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:24:26.191 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install librados2 2024-03-25T18:24:26.762 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:05:35 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:24:26.863 INFO:teuthology.orchestra.run.smithi044.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:24:26.918 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:24:26.919 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-03-25T18:24:26.919 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:24:26.974 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install librbd1 2024-03-25T18:24:27.551 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:05:36 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:24:27.653 INFO:teuthology.orchestra.run.smithi044.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:24:27.710 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:24:27.711 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-03-25T18:24:27.711 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:24:27.763 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install python3-rados 2024-03-25T18:24:28.347 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:05:37 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:24:28.447 INFO:teuthology.orchestra.run.smithi044.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:24:28.502 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:24:28.503 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-03-25T18:24:28.503 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:24:28.555 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install python3-rgw 2024-03-25T18:24:29.126 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:05:38 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:24:29.226 INFO:teuthology.orchestra.run.smithi044.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:24:29.281 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:24:29.282 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-03-25T18:24:29.282 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:24:29.325 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install python3-cephfs 2024-03-25T18:24:29.899 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:05:38 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:24:29.998 INFO:teuthology.orchestra.run.smithi044.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:24:30.052 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:24:30.054 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-03-25T18:24:30.054 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:24:30.115 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install python3-rbd 2024-03-25T18:24:30.683 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:05:39 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:24:30.782 INFO:teuthology.orchestra.run.smithi044.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:24:30.837 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:24:30.838 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-03-25T18:24:30.838 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:24:30.886 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install rbd-fuse 2024-03-25T18:24:31.460 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:05:40 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:24:31.615 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:24:31.616 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:24:31.616 INFO:teuthology.orchestra.run.smithi044.stdout: Package Architecture Version Repository Size 2024-03-25T18:24:31.616 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:24:31.616 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T18:24:31.616 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:24:31.616 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:31.616 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T18:24:31.616 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:24:31.616 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T18:24:31.616 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:31.616 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 85 k 2024-03-25T18:24:31.616 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 231 k 2024-03-25T18:24:31.616 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T18:24:32.319 INFO:teuthology.orchestra.run.smithi044.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 121 kB/s | 85 kB 00:00 2024-03-25T18:24:32.319 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:24:32.320 INFO:teuthology.orchestra.run.smithi044.stdout:Total 121 kB/s | 85 kB 00:00 2024-03-25T18:24:32.320 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T18:24:32.328 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T18:24:32.328 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T18:24:32.387 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T18:24:32.387 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T18:24:32.529 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T18:24:32.620 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:33.003 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:33.330 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:33.330 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:33.330 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T18:24:33.330 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:24:33.330 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:33.330 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:24:33.539 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install rbd-mirror 2024-03-25T18:24:34.113 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:05:43 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:24:34.266 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:24:34.266 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:24:34.266 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T18:24:34.266 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:24:34.267 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T18:24:34.267 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:24:34.267 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:34.267 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T18:24:34.267 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:24:34.267 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T18:24:34.267 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:34.268 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 3.1 M 2024-03-25T18:24:34.268 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 13 M 2024-03-25T18:24:34.268 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T18:24:39.505 INFO:teuthology.orchestra.run.smithi044.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 604 kB/s | 3.1 MB 00:05 2024-03-25T18:24:39.505 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:24:39.506 INFO:teuthology.orchestra.run.smithi044.stdout:Total 604 kB/s | 3.1 MB 00:05 2024-03-25T18:24:39.506 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T18:24:39.518 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T18:24:39.518 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T18:24:39.597 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T18:24:39.598 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T18:24:40.059 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T18:24:40.130 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:40.154 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:40.154 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:24:40.154 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:24:40.154 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:24:40.155 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:24:40.155 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:41.312 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:41.312 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:41.312 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T18:24:41.312 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:24:41.312 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:41.312 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:24:41.524 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install rbd-nbd 2024-03-25T18:24:42.105 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:05:51 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:24:42.260 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:24:42.260 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:24:42.260 INFO:teuthology.orchestra.run.smithi044.stdout: Package Architecture Version Repository Size 2024-03-25T18:24:42.260 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:24:42.260 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T18:24:42.261 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:24:42.261 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:42.261 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T18:24:42.261 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:24:42.261 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T18:24:42.261 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:42.261 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 171 k 2024-03-25T18:24:42.261 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 494 k 2024-03-25T18:24:42.262 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T18:24:42.958 INFO:teuthology.orchestra.run.smithi044.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 246 kB/s | 171 kB 00:00 2024-03-25T18:24:42.959 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:24:42.959 INFO:teuthology.orchestra.run.smithi044.stdout:Total 246 kB/s | 171 kB 00:00 2024-03-25T18:24:42.959 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T18:24:42.968 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T18:24:42.968 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T18:24:43.027 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T18:24:43.028 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T18:24:43.183 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T18:24:43.291 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:43.712 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:44.019 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:44.020 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:44.020 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T18:24:44.020 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:24:44.020 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:44.020 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:24:44.217 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:24:44.792 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:05:53 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:24:44.949 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:24:44.949 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:24:44.949 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T18:24:44.949 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:24:44.949 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T18:24:44.949 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:24:44.950 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-03-25T18:24:44.950 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:24:44.950 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:44.950 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T18:24:44.950 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:24:44.950 INFO:teuthology.orchestra.run.smithi044.stdout:Install 2 Packages 2024-03-25T18:24:44.950 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:24:44.951 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 91 M 2024-03-25T18:24:44.951 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 498 M 2024-03-25T18:24:44.951 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T18:25:04.748 INFO:teuthology.orchestra.run.smithi111.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 545 kB/s | 22 MB 00:42 2024-03-25T18:25:25.591 INFO:teuthology.orchestra.run.smithi044.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 566 kB/s | 22 MB 00:40 2024-03-25T18:25:53.266 INFO:teuthology.orchestra.run.smithi111.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 769 kB/s | 68 MB 01:30 2024-03-25T18:25:53.267 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:25:53.268 INFO:teuthology.orchestra.run.smithi111.stdout:Total 1.0 MB/s | 91 MB 01:30 2024-03-25T18:25:53.268 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T18:25:53.272 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T18:25:53.272 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T18:25:53.853 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T18:25:53.853 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T18:25:56.960 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T18:26:04.067 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:26:04.150 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:26:07.494 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:26:07.494 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:26:07.766 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:26:07.766 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:26:07.766 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T18:26:07.766 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:26:07.766 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:26:07.766 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:26:07.766 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:26:08.050 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:26:08.635 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:06:48 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:26:08.794 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:26:08.795 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:26:08.795 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T18:26:08.795 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:26:08.795 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T18:26:08.796 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:26:08.796 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:26:08.796 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T18:26:08.796 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:26:08.796 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T18:26:08.796 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:26:08.796 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 416 M 2024-03-25T18:26:08.796 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 1.7 G 2024-03-25T18:26:08.796 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T18:26:12.493 INFO:teuthology.orchestra.run.smithi044.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 797 kB/s | 68 MB 01:27 2024-03-25T18:26:12.494 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:26:12.495 INFO:teuthology.orchestra.run.smithi044.stdout:Total 1.0 MB/s | 91 MB 01:27 2024-03-25T18:26:12.495 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T18:26:12.499 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T18:26:12.500 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T18:26:13.111 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T18:26:13.112 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T18:26:16.400 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T18:26:23.930 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:26:23.997 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:26:27.346 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:26:27.347 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:26:27.606 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:26:27.606 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:26:27.606 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T18:26:27.606 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:26:27.606 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:26:27.606 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:26:27.606 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:26:27.812 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:26:28.388 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:07:37 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:26:28.543 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:26:28.544 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:26:28.544 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T18:26:28.544 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:26:28.544 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T18:26:28.544 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:26:28.544 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:26:28.544 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T18:26:28.544 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:26:28.544 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T18:26:28.544 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:26:28.544 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 416 M 2024-03-25T18:26:28.545 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 1.7 G 2024-03-25T18:26:28.545 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T18:36:49.366 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 686 kB/s | 416 MB 10:20 2024-03-25T18:36:49.366 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:36:49.366 INFO:teuthology.orchestra.run.smithi044.stdout:Total 686 kB/s | 416 MB 10:20 2024-03-25T18:36:49.367 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T18:36:49.369 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T18:36:49.369 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T18:36:51.959 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T18:36:51.960 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T18:37:36.678 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T18:37:37.064 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:37:44.163 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:37:44.419 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:37:44.419 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:37:44.419 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T18:37:44.419 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:37:44.419 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:37:44.419 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:37:44.660 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:37:45.244 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:18:54 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:37:45.402 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:37:45.402 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================== 2024-03-25T18:37:45.402 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T18:37:45.402 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================== 2024-03-25T18:37:45.403 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T18:37:45.403 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:37:45.403 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:37:45.403 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T18:37:45.403 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================== 2024-03-25T18:37:45.403 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T18:37:45.403 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:37:45.403 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 1.4 M 2024-03-25T18:37:45.403 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 6.4 M 2024-03-25T18:37:45.403 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T18:37:47.464 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 709 kB/s | 1.4 MB 00:02 2024-03-25T18:37:47.465 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:37:47.465 INFO:teuthology.orchestra.run.smithi044.stdout:Total 708 kB/s | 1.4 MB 00:02 2024-03-25T18:37:47.466 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T18:37:47.470 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T18:37:47.470 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T18:37:47.517 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T18:37:47.517 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T18:37:47.792 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T18:37:47.898 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:37:48.278 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:37:48.579 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:37:48.579 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:37:48.579 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T18:37:48.579 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:37:48.579 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:37:48.579 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:37:48.744 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:37:49.318 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:18:58 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:37:49.474 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:37:49.475 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:37:49.475 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T18:37:49.475 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:37:49.475 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T18:37:49.475 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:37:49.475 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:37:49.475 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T18:37:49.475 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:37:49.475 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T18:37:49.475 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:37:49.475 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 176 M 2024-03-25T18:37:49.475 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 753 M 2024-03-25T18:37:49.475 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T18:38:22.530 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 581 kB/s | 416 MB 12:13 2024-03-25T18:38:22.531 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:38:22.531 INFO:teuthology.orchestra.run.smithi111.stdout:Total 581 kB/s | 416 MB 12:13 2024-03-25T18:38:22.531 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T18:38:22.535 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T18:38:22.535 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T18:38:25.013 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T18:38:25.013 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T18:39:07.184 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T18:39:07.299 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:39:15.354 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:39:15.615 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:39:15.615 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:39:15.615 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T18:39:15.615 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:39:15.616 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:39:15.616 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:39:15.878 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:39:16.445 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:19:56 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:39:16.595 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:39:16.596 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================== 2024-03-25T18:39:16.596 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T18:39:16.596 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================== 2024-03-25T18:39:16.596 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T18:39:16.596 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:39:16.596 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:39:16.597 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T18:39:16.597 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================== 2024-03-25T18:39:16.597 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T18:39:16.597 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:39:16.597 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 1.4 M 2024-03-25T18:39:16.597 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 6.4 M 2024-03-25T18:39:16.597 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T18:39:20.003 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 429 kB/s | 1.4 MB 00:03 2024-03-25T18:39:20.004 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:39:20.004 INFO:teuthology.orchestra.run.smithi111.stdout:Total 429 kB/s | 1.4 MB 00:03 2024-03-25T18:39:20.004 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T18:39:20.008 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T18:39:20.008 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T18:39:20.053 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T18:39:20.054 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T18:39:20.326 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T18:39:20.426 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:39:20.831 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:39:21.116 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:39:21.116 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:39:21.116 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T18:39:21.117 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:39:21.117 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:39:21.117 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:39:21.304 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:39:21.862 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:20:01 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:39:22.009 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:39:22.010 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:39:22.010 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T18:39:22.010 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:39:22.010 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T18:39:22.010 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:39:22.010 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:39:22.010 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T18:39:22.010 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:39:22.011 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T18:39:22.011 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:39:22.011 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 176 M 2024-03-25T18:39:22.011 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 753 M 2024-03-25T18:39:22.011 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T18:43:03.510 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 573 kB/s | 176 MB 05:14 2024-03-25T18:43:03.510 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:43:03.510 INFO:teuthology.orchestra.run.smithi044.stdout:Total 573 kB/s | 176 MB 05:14 2024-03-25T18:43:03.510 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T18:43:03.513 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T18:43:03.513 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T18:43:04.624 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T18:43:04.624 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T18:43:23.703 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T18:43:23.799 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:43:28.467 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:43:28.723 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:43:28.723 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:43:28.723 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T18:43:28.723 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:43:28.723 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:43:28.724 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T18:43:28.914 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-test-debuginfo 2024-03-25T18:43:29.487 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:24:38 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T18:43:29.642 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T18:43:29.643 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:43:29.643 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T18:43:29.643 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:43:29.643 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T18:43:29.643 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T18:43:29.643 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:43:29.643 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T18:43:29.643 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T18:43:29.643 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T18:43:29.643 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T18:43:29.643 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 672 M 2024-03-25T18:43:29.643 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 3.1 G 2024-03-25T18:43:29.643 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T18:48:23.750 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 332 kB/s | 176 MB 09:01 2024-03-25T18:48:23.751 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:48:23.751 INFO:teuthology.orchestra.run.smithi111.stdout:Total 332 kB/s | 176 MB 09:01 2024-03-25T18:48:23.751 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T18:48:23.754 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T18:48:23.754 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T18:48:24.815 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T18:48:24.816 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T18:48:43.020 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T18:48:43.246 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:48:45.681 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:48:45.942 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:48:45.943 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:48:45.943 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T18:48:45.943 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:45.943 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:48:45.943 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T18:48:46.154 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-test-debuginfo 2024-03-25T18:48:46.719 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:29:26 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T18:48:46.867 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T18:48:46.868 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:48:46.868 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T18:48:46.868 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:48:46.868 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T18:48:46.868 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T18:48:46.868 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:48:46.868 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T18:48:46.868 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T18:48:46.869 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T18:48:46.869 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T18:48:46.869 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 672 M 2024-03-25T18:48:46.869 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 3.1 G 2024-03-25T18:48:46.869 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T19:02:43.386 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 597 kB/s | 672 MB 19:13 2024-03-25T19:02:43.386 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:02:43.386 INFO:teuthology.orchestra.run.smithi044.stdout:Total 597 kB/s | 672 MB 19:13 2024-03-25T19:02:43.387 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T19:02:43.391 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T19:02:43.391 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T19:02:47.547 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T19:02:47.547 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T19:04:02.436 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T19:04:02.564 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:04:11.626 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:04:11.886 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:04:11.886 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:04:11.886 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T19:04:11.886 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:04:11.886 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:04:11.887 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T19:04:12.167 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:04:12.746 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:45:21 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T19:04:12.847 INFO:teuthology.orchestra.run.smithi044.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:04:12.902 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T19:04:12.903 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-03-25T19:04:12.903 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T19:04:12.967 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:04:13.542 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:45:22 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T19:04:13.697 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T19:04:13.697 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:04:13.697 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T19:04:13.697 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:04:13.697 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T19:04:13.697 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:04:13.697 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:04:13.698 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T19:04:13.698 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:04:13.698 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T19:04:13.698 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:04:13.698 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 20 M 2024-03-25T19:04:13.698 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 103 M 2024-03-25T19:04:13.698 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T19:04:59.220 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 443 kB/s | 20 MB 00:45 2024-03-25T19:04:59.220 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:04:59.221 INFO:teuthology.orchestra.run.smithi044.stdout:Total 443 kB/s | 20 MB 00:45 2024-03-25T19:04:59.221 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T19:04:59.224 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T19:04:59.224 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T19:04:59.381 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T19:04:59.381 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T19:05:01.739 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T19:05:01.845 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:05:02.850 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:05:03.106 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:05:03.107 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:05:03.107 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T19:05:03.107 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:05:03.107 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:05:03.107 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T19:05:03.271 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:05:03.844 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:46:12 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T19:05:04.000 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T19:05:04.001 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:05:04.001 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T19:05:04.001 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:05:04.001 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T19:05:04.002 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:05:04.002 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:05:04.002 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T19:05:04.002 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:05:04.002 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T19:05:04.002 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:05:04.002 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 48 M 2024-03-25T19:05:04.002 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 173 M 2024-03-25T19:05:04.002 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T19:07:03.284 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 410 kB/s | 48 MB 01:59 2024-03-25T19:07:03.285 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:07:03.285 INFO:teuthology.orchestra.run.smithi044.stdout:Total 410 kB/s | 48 MB 01:59 2024-03-25T19:07:03.285 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T19:07:03.288 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T19:07:03.288 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T19:07:03.615 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T19:07:03.616 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T19:07:08.644 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T19:07:08.735 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:07:10.129 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:07:10.388 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:07:10.389 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:07:10.389 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T19:07:10.389 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:07:10.389 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:07:10.389 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T19:07:10.579 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:07:11.148 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:48:20 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T19:07:11.303 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T19:07:11.304 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:07:11.304 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T19:07:11.304 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:07:11.304 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T19:07:11.304 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:07:11.304 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:07:11.304 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T19:07:11.304 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:07:11.304 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T19:07:11.304 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:07:11.305 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 99 M 2024-03-25T19:07:11.305 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 449 M 2024-03-25T19:07:11.305 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T19:12:14.056 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 333 kB/s | 99 MB 05:02 2024-03-25T19:12:14.056 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:12:14.056 INFO:teuthology.orchestra.run.smithi044.stdout:Total 333 kB/s | 99 MB 05:02 2024-03-25T19:12:14.056 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T19:12:14.059 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T19:12:14.059 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T19:12:14.696 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T19:12:14.697 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T19:12:25.600 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T19:12:25.813 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:12:28.773 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:12:29.030 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:12:29.030 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:12:29.030 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T19:12:29.030 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:12:29.030 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:12:29.030 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T19:12:29.189 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:12:29.765 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:53:38 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T19:12:29.921 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T19:12:29.921 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:12:29.922 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T19:12:29.922 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:12:29.922 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T19:12:29.922 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:12:29.922 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:12:29.922 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T19:12:29.922 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:12:29.922 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T19:12:29.922 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:12:29.922 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 287 M 2024-03-25T19:12:29.922 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 1.2 G 2024-03-25T19:12:29.922 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T19:18:03.596 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 392 kB/s | 672 MB 29:16 2024-03-25T19:18:03.597 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:18:03.597 INFO:teuthology.orchestra.run.smithi111.stdout:Total 392 kB/s | 672 MB 29:16 2024-03-25T19:18:03.597 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T19:18:03.601 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T19:18:03.601 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T19:18:07.576 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T19:18:07.577 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T19:19:18.395 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T19:19:19.022 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:19:27.049 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:19:27.331 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:19:27.331 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:19:27.331 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T19:19:27.331 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:19:27.331 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:19:27.331 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T19:19:27.611 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:19:28.171 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:00:08 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T19:19:28.266 INFO:teuthology.orchestra.run.smithi111.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:19:28.318 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T19:19:28.319 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-25T19:19:28.319 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T19:19:28.404 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:19:28.962 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:00:08 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T19:19:29.109 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T19:19:29.110 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:19:29.110 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T19:19:29.110 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:19:29.110 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T19:19:29.110 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:19:29.110 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:19:29.110 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T19:19:29.110 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:19:29.110 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T19:19:29.110 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:19:29.111 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 20 M 2024-03-25T19:19:29.111 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 103 M 2024-03-25T19:19:29.111 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T19:20:01.270 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 627 kB/s | 20 MB 00:32 2024-03-25T19:20:01.271 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:20:01.271 INFO:teuthology.orchestra.run.smithi111.stdout:Total 627 kB/s | 20 MB 00:32 2024-03-25T19:20:01.272 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T19:20:01.276 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T19:20:01.276 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T19:20:01.426 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T19:20:01.426 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T19:20:03.649 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T19:20:03.959 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:20:04.946 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:20:05.225 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:20:05.225 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:20:05.225 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T19:20:05.226 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:20:05.226 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:20:05.226 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T19:20:05.409 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:20:05.971 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:00:45 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T19:20:06.120 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T19:20:06.120 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:20:06.120 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T19:20:06.120 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:20:06.121 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T19:20:06.121 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:20:06.121 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:20:06.121 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T19:20:06.121 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:20:06.121 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T19:20:06.121 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:20:06.121 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 48 M 2024-03-25T19:20:06.121 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 173 M 2024-03-25T19:20:06.122 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T19:21:23.688 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 630 kB/s | 48 MB 01:17 2024-03-25T19:21:23.689 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:21:23.689 INFO:teuthology.orchestra.run.smithi111.stdout:Total 630 kB/s | 48 MB 01:17 2024-03-25T19:21:23.689 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T19:21:23.692 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T19:21:23.692 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T19:21:24.004 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T19:21:24.004 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T19:21:28.759 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T19:21:28.869 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:21:30.237 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:21:30.524 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:21:30.524 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:21:30.524 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T19:21:30.524 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:21:30.524 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:21:30.524 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T19:21:30.727 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:21:31.320 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:02:11 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T19:21:31.476 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T19:21:31.477 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:21:31.477 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T19:21:31.477 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:21:31.477 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T19:21:31.477 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:21:31.477 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:21:31.477 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T19:21:31.477 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:21:31.477 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T19:21:31.477 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:21:31.477 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 99 M 2024-03-25T19:21:31.477 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 449 M 2024-03-25T19:21:31.478 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T19:23:37.694 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 440 kB/s | 287 MB 11:07 2024-03-25T19:23:37.694 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:23:37.695 INFO:teuthology.orchestra.run.smithi044.stdout:Total 440 kB/s | 287 MB 11:07 2024-03-25T19:23:37.695 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T19:23:37.698 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T19:23:37.698 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T19:23:39.488 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T19:23:39.488 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T19:24:10.602 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T19:24:10.709 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:24:11.990 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:24:12.291 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:24:12.291 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:24:12.291 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T19:24:12.291 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:24:12.291 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:24:12.291 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T19:24:12.486 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:24:13.060 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 1:05:22 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T19:24:13.214 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T19:24:13.214 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:24:13.215 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T19:24:13.215 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:24:13.215 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T19:24:13.215 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:24:13.215 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:24:13.215 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T19:24:13.215 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:24:13.215 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T19:24:13.215 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:24:13.215 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 5.9 M 2024-03-25T19:24:13.215 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 28 M 2024-03-25T19:24:13.215 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T19:24:24.267 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 584 kB/s | 99 MB 02:52 2024-03-25T19:24:24.268 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:24.268 INFO:teuthology.orchestra.run.smithi111.stdout:Total 584 kB/s | 99 MB 02:52 2024-03-25T19:24:24.268 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T19:24:24.272 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T19:24:24.272 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T19:24:24.883 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T19:24:24.883 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T19:24:26.434 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 460 kB/s | 5.9 MB 00:13 2024-03-25T19:24:26.435 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:26.435 INFO:teuthology.orchestra.run.smithi044.stdout:Total 460 kB/s | 5.9 MB 00:13 2024-03-25T19:24:26.436 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T19:24:26.440 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T19:24:26.440 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T19:24:26.515 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T19:24:26.515 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T19:24:27.292 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T19:24:27.402 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:24:27.968 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:24:28.242 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:24:28.242 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:24:28.242 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T19:24:28.242 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:24:28.242 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:24:28.242 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T19:24:28.466 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:24:29.041 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 1:05:38 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T19:24:29.198 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T19:24:29.198 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:24:29.199 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T19:24:29.199 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:24:29.199 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T19:24:29.199 INFO:teuthology.orchestra.run.smithi044.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:24:29.199 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:24:29.199 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T19:24:29.199 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:24:29.199 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T19:24:29.199 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:24:29.200 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 109 k 2024-03-25T19:24:29.200 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 440 k 2024-03-25T19:24:29.200 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T19:24:29.878 INFO:teuthology.orchestra.run.smithi044.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 160 kB/s | 109 kB 00:00 2024-03-25T19:24:29.879 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:29.879 INFO:teuthology.orchestra.run.smithi044.stdout:Total 160 kB/s | 109 kB 00:00 2024-03-25T19:24:29.880 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T19:24:29.883 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T19:24:29.883 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T19:24:29.922 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T19:24:29.922 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T19:24:30.042 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T19:24:30.163 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:24:30.579 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:24:30.848 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:24:30.848 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:24:30.848 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T19:24:30.848 INFO:teuthology.orchestra.run.smithi044.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:24:30.849 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:24:30.849 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T19:24:31.027 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:24:31.601 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 1:05:40 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T19:24:31.756 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T19:24:31.756 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:24:31.757 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T19:24:31.757 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:24:31.757 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T19:24:31.757 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:24:31.757 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:24:31.757 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T19:24:31.757 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:24:31.757 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T19:24:31.757 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:24:31.758 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 5.4 M 2024-03-25T19:24:31.758 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 25 M 2024-03-25T19:24:31.758 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T19:24:35.161 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T19:24:35.273 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:24:38.190 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:24:38.442 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:24:38.442 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:24:38.442 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T19:24:38.442 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:24:38.442 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:24:38.442 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T19:24:38.633 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:24:39.198 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:05:19 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T19:24:39.347 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T19:24:39.347 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:24:39.347 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T19:24:39.347 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:24:39.348 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T19:24:39.348 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:24:39.348 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:24:39.348 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T19:24:39.348 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:24:39.348 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T19:24:39.348 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:24:39.349 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 287 M 2024-03-25T19:24:39.349 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 1.2 G 2024-03-25T19:24:39.349 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T19:24:45.480 INFO:teuthology.orchestra.run.smithi044.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 399 kB/s | 5.4 MB 00:13 2024-03-25T19:24:45.481 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:45.481 INFO:teuthology.orchestra.run.smithi044.stdout:Total 399 kB/s | 5.4 MB 00:13 2024-03-25T19:24:45.481 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T19:24:45.486 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T19:24:45.486 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T19:24:45.556 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T19:24:45.556 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T19:24:46.260 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T19:24:46.365 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:24:46.947 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:24:47.213 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:24:47.214 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:24:47.214 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T19:24:47.214 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:24:47.214 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:24:47.214 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T19:24:47.422 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install librados2-debuginfo 2024-03-25T19:24:47.994 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 1:05:56 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T19:24:48.149 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T19:24:48.150 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:24:48.150 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T19:24:48.150 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:24:48.150 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T19:24:48.150 INFO:teuthology.orchestra.run.smithi044.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:24:48.150 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:24:48.151 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T19:24:48.151 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:24:48.151 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T19:24:48.151 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:24:48.151 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 44 M 2024-03-25T19:24:48.151 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 253 M 2024-03-25T19:24:48.151 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T19:25:56.101 INFO:teuthology.orchestra.run.smithi044.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 667 kB/s | 44 MB 01:07 2024-03-25T19:25:56.102 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:56.102 INFO:teuthology.orchestra.run.smithi044.stdout:Total 667 kB/s | 44 MB 01:07 2024-03-25T19:25:56.103 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T19:25:56.106 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T19:25:56.106 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T19:25:56.422 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T19:25:56.422 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T19:26:01.805 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T19:26:01.902 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:26:03.755 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:26:04.024 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:26:04.024 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:26:04.024 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T19:26:04.024 INFO:teuthology.orchestra.run.smithi044.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:26:04.024 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:26:04.024 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T19:26:04.225 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:26:04.801 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 1:07:13 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T19:26:04.958 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T19:26:04.958 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:26:04.958 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T19:26:04.959 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:26:04.959 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T19:26:04.959 INFO:teuthology.orchestra.run.smithi044.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:26:04.959 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:26:04.959 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T19:26:04.959 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:26:04.959 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T19:26:04.959 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:26:04.959 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 52 M 2024-03-25T19:26:04.959 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 308 M 2024-03-25T19:26:04.959 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T19:27:44.578 INFO:teuthology.orchestra.run.smithi044.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 533 kB/s | 52 MB 01:39 2024-03-25T19:27:44.578 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:27:44.579 INFO:teuthology.orchestra.run.smithi044.stdout:Total 533 kB/s | 52 MB 01:39 2024-03-25T19:27:44.579 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T19:27:44.582 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T19:27:44.582 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T19:27:44.942 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T19:27:44.943 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T19:27:51.307 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T19:27:51.412 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:27:53.598 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:27:53.865 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:27:53.865 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:27:53.865 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T19:27:53.865 INFO:teuthology.orchestra.run.smithi044.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:27:53.865 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:27:53.866 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T19:27:54.058 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:27:54.627 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 1:09:03 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T19:27:54.782 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T19:27:54.782 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:27:54.783 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T19:27:54.783 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:27:54.783 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T19:27:54.783 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:27:54.783 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:27:54.783 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T19:27:54.783 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:27:54.783 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T19:27:54.783 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:27:54.783 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 615 k 2024-03-25T19:27:54.783 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 2.1 M 2024-03-25T19:27:54.783 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T19:27:56.729 INFO:teuthology.orchestra.run.smithi044.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 316 kB/s | 615 kB 00:01 2024-03-25T19:27:56.730 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:27:56.730 INFO:teuthology.orchestra.run.smithi044.stdout:Total 316 kB/s | 615 kB 00:01 2024-03-25T19:27:56.731 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T19:27:56.735 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T19:27:56.735 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T19:27:56.779 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T19:27:56.779 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T19:27:56.954 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T19:27:57.041 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:27:57.459 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:27:57.719 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:27:57.719 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:27:57.719 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T19:27:57.719 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:27:57.719 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:27:57.719 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T19:27:57.914 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:27:58.488 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 1:09:07 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T19:27:58.648 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T19:27:58.648 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:27:58.648 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T19:27:58.648 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:27:58.648 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T19:27:58.648 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:27:58.648 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:27:58.648 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T19:27:58.648 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:27:58.649 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T19:27:58.649 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:27:58.649 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 771 k 2024-03-25T19:27:58.649 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 2.7 M 2024-03-25T19:27:58.649 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T19:28:00.666 INFO:teuthology.orchestra.run.smithi044.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 382 kB/s | 771 kB 00:02 2024-03-25T19:28:00.667 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:28:00.667 INFO:teuthology.orchestra.run.smithi044.stdout:Total 382 kB/s | 771 kB 00:02 2024-03-25T19:28:00.667 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T19:28:00.671 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T19:28:00.672 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T19:28:00.716 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T19:28:00.716 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T19:28:00.913 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T19:28:01.021 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:28:01.399 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:28:01.713 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:28:01.713 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:28:01.713 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T19:28:01.713 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:28:01.713 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:28:01.714 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T19:28:01.914 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:28:02.490 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 1:09:11 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T19:28:02.645 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T19:28:02.645 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:28:02.646 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T19:28:02.646 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:28:02.646 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T19:28:02.646 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:28:02.646 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:28:02.646 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T19:28:02.646 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:28:02.646 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T19:28:02.646 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:28:02.646 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 1.1 M 2024-03-25T19:28:02.646 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 4.3 M 2024-03-25T19:28:02.646 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T19:28:04.494 INFO:teuthology.orchestra.run.smithi044.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 623 kB/s | 1.1 MB 00:01 2024-03-25T19:28:04.495 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:28:04.495 INFO:teuthology.orchestra.run.smithi044.stdout:Total 622 kB/s | 1.1 MB 00:01 2024-03-25T19:28:04.495 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T19:28:04.499 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T19:28:04.499 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T19:28:04.545 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T19:28:04.546 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T19:28:04.775 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T19:28:04.896 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:28:05.317 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:28:05.588 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:28:05.588 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:28:05.588 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T19:28:05.589 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:28:05.589 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:28:05.589 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T19:28:05.792 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:28:06.364 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 1:09:15 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T19:28:06.519 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T19:28:06.519 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:28:06.519 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T19:28:06.519 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:28:06.520 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T19:28:06.520 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:28:06.520 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:28:06.520 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T19:28:06.520 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:28:06.520 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T19:28:06.520 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:28:06.520 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 337 k 2024-03-25T19:28:06.520 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 1.0 M 2024-03-25T19:28:06.520 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T19:28:07.305 INFO:teuthology.orchestra.run.smithi044.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 429 kB/s | 337 kB 00:00 2024-03-25T19:28:07.306 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:28:07.306 INFO:teuthology.orchestra.run.smithi044.stdout:Total 429 kB/s | 337 kB 00:00 2024-03-25T19:28:07.306 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T19:28:07.310 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T19:28:07.310 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T19:28:07.350 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T19:28:07.351 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T19:28:07.498 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T19:28:07.608 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:28:07.994 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:28:08.263 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:28:08.263 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:28:08.263 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T19:28:08.263 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:28:08.263 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:28:08.263 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T19:28:08.474 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:28:09.051 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 1:09:18 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T19:28:09.205 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T19:28:09.206 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:28:09.206 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T19:28:09.206 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:28:09.206 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T19:28:09.206 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:28:09.206 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:28:09.206 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T19:28:09.206 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:28:09.206 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T19:28:09.206 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:28:09.206 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 509 k 2024-03-25T19:28:09.207 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 2.5 M 2024-03-25T19:28:09.207 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T19:28:10.567 INFO:teuthology.orchestra.run.smithi044.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 374 kB/s | 509 kB 00:01 2024-03-25T19:28:10.568 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:28:10.568 INFO:teuthology.orchestra.run.smithi044.stdout:Total 374 kB/s | 509 kB 00:01 2024-03-25T19:28:10.568 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T19:28:10.572 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T19:28:10.572 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T19:28:10.615 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T19:28:10.616 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T19:28:10.797 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T19:28:10.911 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:28:11.284 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:28:11.563 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:28:11.563 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:28:11.563 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T19:28:11.563 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:28:11.563 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:28:11.563 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T19:28:11.781 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:28:12.357 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 1:09:21 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T19:28:12.513 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T19:28:12.513 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:28:12.513 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T19:28:12.513 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:28:12.513 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T19:28:12.513 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:28:12.513 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:28:12.514 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T19:28:12.514 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:28:12.514 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T19:28:12.514 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:28:12.514 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 52 M 2024-03-25T19:28:12.514 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 298 M 2024-03-25T19:28:12.514 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T19:28:51.900 INFO:teuthology.orchestra.run.smithi044.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 1.3 MB/s | 52 MB 00:39 2024-03-25T19:28:51.900 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:28:51.901 INFO:teuthology.orchestra.run.smithi044.stdout:Total 1.3 MB/s | 52 MB 00:39 2024-03-25T19:28:51.901 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T19:28:51.905 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T19:28:51.905 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T19:28:52.257 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T19:28:52.257 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T19:28:58.511 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T19:28:58.618 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:29:00.823 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:29:01.097 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:29:01.097 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:29:01.097 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T19:29:01.097 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:29:01.098 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:29:01.098 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T19:29:01.316 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:29:01.889 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 1:10:10 ago on Mon 25 Mar 2024 06:18:51 PM UTC. 2024-03-25T19:29:02.045 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-03-25T19:29:02.046 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:29:02.046 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repo Size 2024-03-25T19:29:02.046 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:29:02.046 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-03-25T19:29:02.046 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:29:02.046 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:29:02.047 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-03-25T19:29:02.047 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-03-25T19:29:02.047 INFO:teuthology.orchestra.run.smithi044.stdout:Install 1 Package 2024-03-25T19:29:02.047 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:29:02.047 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 1.3 M 2024-03-25T19:29:02.047 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 5.5 M 2024-03-25T19:29:02.047 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-03-25T19:29:03.785 INFO:teuthology.orchestra.run.smithi044.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 738 kB/s | 1.3 MB 00:01 2024-03-25T19:29:03.786 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:29:03.786 INFO:teuthology.orchestra.run.smithi044.stdout:Total 737 kB/s | 1.3 MB 00:01 2024-03-25T19:29:03.787 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-03-25T19:29:03.791 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-03-25T19:29:03.791 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-03-25T19:29:03.838 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-03-25T19:29:03.839 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-03-25T19:29:04.091 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-03-25T19:29:04.187 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:29:04.561 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:29:04.840 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:29:04.840 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:29:04.840 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-03-25T19:29:04.840 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:29:04.840 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:29:04.840 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-03-25T19:29:05.037 DEBUG:teuthology.parallel:result is None 2024-03-25T19:31:08.950 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 754 kB/s | 287 MB 06:29 2024-03-25T19:31:08.950 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:31:08.951 INFO:teuthology.orchestra.run.smithi111.stdout:Total 754 kB/s | 287 MB 06:29 2024-03-25T19:31:08.951 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T19:31:08.954 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T19:31:08.954 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T19:31:10.666 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T19:31:10.666 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T19:31:39.985 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T19:31:40.078 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:31:47.754 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:31:48.032 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:31:48.032 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:31:48.032 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T19:31:48.032 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:31:48.032 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:31:48.032 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T19:31:48.250 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:31:48.817 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:12:28 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T19:31:48.966 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T19:31:48.967 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:31:48.967 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T19:31:48.967 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:31:48.967 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T19:31:48.967 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:31:48.968 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:31:48.968 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T19:31:48.968 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:31:48.968 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T19:31:48.968 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:31:48.968 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 5.9 M 2024-03-25T19:31:48.968 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 28 M 2024-03-25T19:31:48.968 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T19:31:52.598 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 1.6 MB/s | 5.9 MB 00:03 2024-03-25T19:31:52.599 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:31:52.599 INFO:teuthology.orchestra.run.smithi111.stdout:Total 1.6 MB/s | 5.9 MB 00:03 2024-03-25T19:31:52.600 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T19:31:52.604 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T19:31:52.604 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T19:31:52.676 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T19:31:52.677 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T19:31:53.417 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T19:31:53.522 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:31:54.087 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:31:54.368 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:31:54.368 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:31:54.368 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T19:31:54.368 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:31:54.368 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:31:54.368 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T19:31:54.583 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:31:55.141 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:12:35 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T19:31:55.292 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T19:31:55.293 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:31:55.293 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T19:31:55.293 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:31:55.293 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T19:31:55.293 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:31:55.293 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:31:55.293 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T19:31:55.294 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:31:55.294 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T19:31:55.294 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:31:55.294 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 109 k 2024-03-25T19:31:55.294 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 440 k 2024-03-25T19:31:55.294 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T19:31:55.976 INFO:teuthology.orchestra.run.smithi111.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 159 kB/s | 109 kB 00:00 2024-03-25T19:31:55.976 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:31:55.977 INFO:teuthology.orchestra.run.smithi111.stdout:Total 159 kB/s | 109 kB 00:00 2024-03-25T19:31:55.977 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T19:31:55.980 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T19:31:55.980 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T19:31:56.015 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T19:31:56.015 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T19:31:56.128 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T19:31:56.241 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:31:56.610 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:31:56.877 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:31:56.877 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:31:56.877 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T19:31:56.877 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:31:56.877 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:31:56.877 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T19:31:57.067 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:31:57.626 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:12:37 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T19:31:57.775 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T19:31:57.776 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:31:57.776 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T19:31:57.776 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:31:57.776 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T19:31:57.776 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:31:57.776 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:31:57.776 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T19:31:57.776 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:31:57.777 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T19:31:57.777 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:31:57.777 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 5.4 M 2024-03-25T19:31:57.777 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 25 M 2024-03-25T19:31:57.777 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T19:32:01.603 INFO:teuthology.orchestra.run.smithi111.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 1.4 MB/s | 5.4 MB 00:03 2024-03-25T19:32:01.604 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:32:01.604 INFO:teuthology.orchestra.run.smithi111.stdout:Total 1.4 MB/s | 5.4 MB 00:03 2024-03-25T19:32:01.605 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T19:32:01.609 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T19:32:01.609 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T19:32:01.677 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T19:32:01.678 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T19:32:02.339 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T19:32:02.443 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:32:03.028 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:32:03.291 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:32:03.292 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:32:03.292 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T19:32:03.292 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:32:03.292 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:32:03.292 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T19:32:03.505 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librados2-debuginfo 2024-03-25T19:32:04.068 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:12:44 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T19:32:04.216 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T19:32:04.216 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:32:04.216 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T19:32:04.217 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:32:04.217 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T19:32:04.217 INFO:teuthology.orchestra.run.smithi111.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:32:04.217 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:32:04.217 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T19:32:04.217 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:32:04.217 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T19:32:04.217 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:32:04.217 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 44 M 2024-03-25T19:32:04.217 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 253 M 2024-03-25T19:32:04.217 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T19:32:39.169 INFO:teuthology.orchestra.run.smithi111.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 1.3 MB/s | 44 MB 00:34 2024-03-25T19:32:39.170 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:32:39.170 INFO:teuthology.orchestra.run.smithi111.stdout:Total 1.3 MB/s | 44 MB 00:34 2024-03-25T19:32:39.170 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T19:32:39.173 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T19:32:39.173 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T19:32:39.471 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T19:32:39.471 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T19:32:44.503 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T19:32:44.600 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:32:46.503 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:32:46.771 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:32:46.771 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:32:46.771 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T19:32:46.771 INFO:teuthology.orchestra.run.smithi111.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:32:46.771 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:32:46.771 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T19:32:46.975 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:32:47.540 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:13:27 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T19:32:47.700 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T19:32:47.701 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:32:47.701 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T19:32:47.701 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:32:47.701 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T19:32:47.701 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:32:47.701 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:32:47.701 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T19:32:47.701 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:32:47.702 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T19:32:47.702 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:32:47.702 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 52 M 2024-03-25T19:32:47.702 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 308 M 2024-03-25T19:32:47.702 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T19:33:22.292 INFO:teuthology.orchestra.run.smithi111.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 1.5 MB/s | 52 MB 00:34 2024-03-25T19:33:22.293 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:22.293 INFO:teuthology.orchestra.run.smithi111.stdout:Total 1.5 MB/s | 52 MB 00:34 2024-03-25T19:33:22.294 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T19:33:22.297 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T19:33:22.297 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T19:33:22.640 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T19:33:22.641 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T19:33:28.638 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T19:33:28.732 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:33:30.867 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:33:31.135 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:33:31.135 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:33:31.136 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T19:33:31.136 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:31.136 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:33:31.136 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T19:33:31.349 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:33:31.906 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:14:11 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T19:33:32.055 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T19:33:32.056 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:33:32.056 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T19:33:32.056 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:33:32.056 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T19:33:32.056 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:33:32.056 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:33:32.056 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T19:33:32.056 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:33:32.056 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T19:33:32.057 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:33:32.057 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 615 k 2024-03-25T19:33:32.057 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 2.1 M 2024-03-25T19:33:32.057 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T19:33:33.669 INFO:teuthology.orchestra.run.smithi111.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 381 kB/s | 615 kB 00:01 2024-03-25T19:33:33.669 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:33.670 INFO:teuthology.orchestra.run.smithi111.stdout:Total 381 kB/s | 615 kB 00:01 2024-03-25T19:33:33.670 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T19:33:33.674 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T19:33:33.674 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T19:33:33.715 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T19:33:33.716 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T19:33:33.883 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T19:33:33.993 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:33:34.367 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:33:34.630 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:33:34.630 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:33:34.630 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T19:33:34.630 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:34.630 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:33:34.630 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T19:33:34.853 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:33:35.413 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:14:15 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T19:33:35.564 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T19:33:35.565 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:33:35.565 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T19:33:35.565 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:33:35.566 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T19:33:35.566 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:33:35.566 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:33:35.566 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T19:33:35.566 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:33:35.566 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T19:33:35.566 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:33:35.566 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 771 k 2024-03-25T19:33:35.566 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 2.7 M 2024-03-25T19:33:35.566 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T19:33:37.189 INFO:teuthology.orchestra.run.smithi111.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 475 kB/s | 771 kB 00:01 2024-03-25T19:33:37.189 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:37.190 INFO:teuthology.orchestra.run.smithi111.stdout:Total 475 kB/s | 771 kB 00:01 2024-03-25T19:33:37.190 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T19:33:37.194 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T19:33:37.194 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T19:33:37.237 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T19:33:37.237 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T19:33:37.423 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T19:33:37.521 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:33:37.895 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:33:38.144 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:33:38.144 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:33:38.145 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T19:33:38.145 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:38.145 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:33:38.145 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T19:33:38.378 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:33:38.953 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:14:18 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T19:33:39.113 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T19:33:39.113 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:33:39.113 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T19:33:39.114 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:33:39.114 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T19:33:39.114 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:33:39.114 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:33:39.114 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T19:33:39.114 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:33:39.114 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T19:33:39.114 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:33:39.114 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 1.1 M 2024-03-25T19:33:39.114 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 4.3 M 2024-03-25T19:33:39.114 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T19:33:41.750 INFO:teuthology.orchestra.run.smithi111.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 437 kB/s | 1.1 MB 00:02 2024-03-25T19:33:41.751 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:41.751 INFO:teuthology.orchestra.run.smithi111.stdout:Total 436 kB/s | 1.1 MB 00:02 2024-03-25T19:33:41.751 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T19:33:41.755 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T19:33:41.755 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T19:33:41.798 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T19:33:41.799 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T19:33:42.018 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T19:33:42.107 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:33:42.466 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:33:42.727 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:33:42.727 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:33:42.727 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T19:33:42.727 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:42.727 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:33:42.727 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T19:33:42.955 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:33:43.512 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:14:23 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T19:33:43.659 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T19:33:43.660 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:33:43.660 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T19:33:43.660 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:33:43.660 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T19:33:43.660 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:33:43.660 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:33:43.660 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T19:33:43.661 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:33:43.661 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T19:33:43.661 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:33:43.661 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 337 k 2024-03-25T19:33:43.661 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 1.0 M 2024-03-25T19:33:43.661 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T19:33:45.703 INFO:teuthology.orchestra.run.smithi111.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 165 kB/s | 337 kB 00:02 2024-03-25T19:33:45.703 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:45.704 INFO:teuthology.orchestra.run.smithi111.stdout:Total 165 kB/s | 337 kB 00:02 2024-03-25T19:33:45.704 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T19:33:45.708 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T19:33:45.708 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T19:33:45.747 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T19:33:45.748 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T19:33:45.888 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T19:33:46.001 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:33:46.413 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:33:46.679 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:33:46.894 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:33:46.894 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T19:33:46.894 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:46.895 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:33:46.895 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T19:33:46.896 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:33:47.461 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:14:27 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T19:33:47.609 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T19:33:47.610 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:33:47.610 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T19:33:47.610 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:33:47.610 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T19:33:47.610 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:33:47.610 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:33:47.610 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T19:33:47.610 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:33:47.610 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T19:33:47.611 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:33:47.611 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 509 k 2024-03-25T19:33:47.611 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 2.5 M 2024-03-25T19:33:47.611 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T19:33:49.073 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 348 kB/s | 509 kB 00:01 2024-03-25T19:33:49.073 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:49.074 INFO:teuthology.orchestra.run.smithi111.stdout:Total 348 kB/s | 509 kB 00:01 2024-03-25T19:33:49.074 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T19:33:49.078 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T19:33:49.078 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T19:33:49.119 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T19:33:49.119 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T19:33:49.276 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T19:33:49.371 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:33:49.748 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:33:50.037 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:33:50.037 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:33:50.037 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T19:33:50.037 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:50.037 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:33:50.037 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T19:33:50.237 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:33:50.803 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:14:30 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T19:33:50.951 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T19:33:50.952 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:33:50.953 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T19:33:50.953 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:33:50.953 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T19:33:50.953 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:33:50.953 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:33:50.953 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T19:33:50.953 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:33:50.953 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T19:33:50.953 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:33:50.953 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 52 M 2024-03-25T19:33:50.953 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 298 M 2024-03-25T19:33:50.954 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T19:34:19.614 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 1.8 MB/s | 52 MB 00:28 2024-03-25T19:34:19.614 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:34:19.614 INFO:teuthology.orchestra.run.smithi111.stdout:Total 1.8 MB/s | 52 MB 00:28 2024-03-25T19:34:19.615 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T19:34:19.618 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T19:34:19.618 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T19:34:19.955 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T19:34:19.955 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T19:34:25.865 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T19:34:25.962 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:34:28.003 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:34:28.260 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:34:28.260 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:28.260 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T19:34:28.260 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:34:28.260 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:28.260 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T19:34:28.560 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:34:29.121 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 1:15:09 ago on Mon 25 Mar 2024 06:19:20 PM UTC. 2024-03-25T19:34:29.269 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-25T19:34:29.269 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:34:29.269 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-25T19:34:29.269 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:34:29.270 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-25T19:34:29.270 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:34:29.270 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:29.270 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-25T19:34:29.270 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-25T19:34:29.270 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-25T19:34:29.270 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:29.270 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 1.3 M 2024-03-25T19:34:29.271 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 5.5 M 2024-03-25T19:34:29.271 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-25T19:34:31.882 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 491 kB/s | 1.3 MB 00:02 2024-03-25T19:34:31.883 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:34:31.883 INFO:teuthology.orchestra.run.smithi111.stdout:Total 491 kB/s | 1.3 MB 00:02 2024-03-25T19:34:31.884 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-25T19:34:31.888 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-25T19:34:31.888 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-25T19:34:31.934 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-25T19:34:31.934 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-25T19:34:32.174 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-25T19:34:32.273 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:34:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:34:32.955 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:34:32.955 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:32.955 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-25T19:34:32.955 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:34:32.956 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:32.956 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-25T19:34:33.210 DEBUG:teuthology.parallel:result is None 2024-03-25T19:34:33.211 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:34:33.354 DEBUG:teuthology.orchestra.run.smithi044:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:34:33.379 INFO:teuthology.orchestra.run.smithi044.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:34:33.379 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:34:33.379 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:34:33.381 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:34:33.520 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:34:33.541 INFO:teuthology.orchestra.run.smithi111.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:34:33.542 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:34:33.542 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:34:33.543 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T19:34:33.543 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T19:34:33.543 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:34:33.575 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T19:34:33.575 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:34:33.609 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T19:34:33.609 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T19:34:33.609 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:34:33.644 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:34:33.714 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T19:34:33.714 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:34:33.741 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:34:33.808 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T19:34:33.809 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T19:34:33.809 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:34:33.838 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:34:33.908 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T19:34:33.908 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:34:33.934 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:34:34.001 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T19:34:34.001 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T19:34:34.001 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:34:34.030 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:34:34.099 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T19:34:34.099 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:34:34.125 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:34:34.192 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T19:34:34.276 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T19:34:34.277 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 777 /var/log/ceph 2024-03-25T19:34:34.279 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 777 /var/log/ceph 2024-03-25T19:34:34.306 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T19:34:34.306 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:34:34.348 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:34:34.374 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T19:34:34.375 DEBUG:teuthology.orchestra.run.smithi044:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:34:34.416 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:34:34.453 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T19:34:34.453 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T19:34:34.453 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621321', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621321', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi044.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJiZboMgAKUllmKZXbQtFQg4Vo+6OojgRvyCN6U7ovHyVJwE05w3X98t/cBtzDQ24ni5kRFzxbZOCHzqX/RoBeY=', 'smithi111.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB6T5MuIscewO93LAeMlMw2BXb8rpyN4fq4Q2BO2Ncz/pFnpvRkgRfFEX/tARk77Vj9MbOnGxQm1YbS2Y7/PybQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T19:34:34.454 DEBUG:teuthology.orchestra.run.smithi044:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:34:34.484 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:34:34.498 DEBUG:teuthology.orchestra.run.smithi044:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:34:34.539 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:34:34.565 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T19:34:34.565 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:34:34.619 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:34:34.619 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_1 2024-03-25T19:34:34.673 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:34:34.673 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:34:34.673 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-25T19:34:34.674 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:34:34.674 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:34:34.674 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-03-25 18:24:40.409933153 +0000 2024-03-25T19:34:34.674 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-03-25 18:15:46.177009246 +0000 2024-03-25T19:34:34.674 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-03-25 18:15:46.177009246 +0000 2024-03-25T19:34:34.674 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-03-25 18:15:46.177009246 +0000 2024-03-25T19:34:34.674 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:34:34.739 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-03-25T19:34:34.740 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-03-25T19:34:34.740 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000277337 s, 1.8 MB/s 2024-03-25T19:34:34.741 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:34:34.796 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_2 2024-03-25T19:34:34.851 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:34:34.851 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:34:34.851 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-25T19:34:34.851 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:34:34.851 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:34:34.851 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-03-25 18:24:40.410933136 +0000 2024-03-25T19:34:34.851 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-03-25 18:15:46.503003729 +0000 2024-03-25T19:34:34.851 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-03-25 18:15:46.503003729 +0000 2024-03-25T19:34:34.851 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-03-25 18:15:46.503003729 +0000 2024-03-25T19:34:34.851 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:34:34.915 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-03-25T19:34:34.915 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-03-25T19:34:34.915 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000192659 s, 2.7 MB/s 2024-03-25T19:34:34.917 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:34:34.971 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_3 2024-03-25T19:34:35.026 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:34:35.026 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:34:35.026 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T19:34:35.026 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:34:35.026 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:34:35.026 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-03-25 18:24:40.410933136 +0000 2024-03-25T19:34:35.026 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-03-25 18:15:46.836998076 +0000 2024-03-25T19:34:35.026 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-03-25 18:15:46.836998076 +0000 2024-03-25T19:34:35.027 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-03-25 18:15:46.836998076 +0000 2024-03-25T19:34:35.027 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:34:35.092 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-03-25T19:34:35.092 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-03-25T19:34:35.092 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000181818 s, 2.8 MB/s 2024-03-25T19:34:35.093 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:34:35.148 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_4 2024-03-25T19:34:35.202 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:34:35.202 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:34:35.202 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-25T19:34:35.202 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:34:35.203 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:34:35.203 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-03-25 18:24:40.410933136 +0000 2024-03-25T19:34:35.203 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-03-25 18:15:47.173992373 +0000 2024-03-25T19:34:35.203 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-03-25 18:15:47.173992373 +0000 2024-03-25T19:34:35.203 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-03-25 18:15:47.173992373 +0000 2024-03-25T19:34:35.203 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:34:35.267 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-03-25T19:34:35.267 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-03-25T19:34:35.267 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000198772 s, 2.6 MB/s 2024-03-25T19:34:35.269 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:34:35.323 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:34:35.324 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T19:34:35.324 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:34:35.340 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:34:35.340 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_1 2024-03-25T19:34:35.396 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:34:35.396 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:34:35.396 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-25T19:34:35.396 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:34:35.397 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:34:35.397 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-03-25 18:24:17.712117006 +0000 2024-03-25T19:34:35.397 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-03-25 18:15:45.662713031 +0000 2024-03-25T19:34:35.397 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-03-25 18:15:45.662713031 +0000 2024-03-25T19:34:35.397 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-03-25 18:15:45.662713031 +0000 2024-03-25T19:34:35.397 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:34:35.463 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-03-25T19:34:35.463 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-03-25T19:34:35.463 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000219232 s, 2.3 MB/s 2024-03-25T19:34:35.464 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:34:35.521 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_2 2024-03-25T19:34:35.575 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:34:35.575 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:34:35.576 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-25T19:34:35.576 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:34:35.576 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:34:35.576 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-03-25 18:24:17.713116988 +0000 2024-03-25T19:34:35.576 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-03-25 18:15:46.000707060 +0000 2024-03-25T19:34:35.576 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-03-25 18:15:46.000707060 +0000 2024-03-25T19:34:35.576 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-03-25 18:15:46.000707060 +0000 2024-03-25T19:34:35.576 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:34:35.641 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-03-25T19:34:35.641 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-03-25T19:34:35.641 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000176771 s, 2.9 MB/s 2024-03-25T19:34:35.642 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:34:35.698 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_3 2024-03-25T19:34:35.753 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:34:35.753 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:34:35.753 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T19:34:35.753 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:34:35.753 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:34:35.753 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-03-25 18:24:17.713116988 +0000 2024-03-25T19:34:35.753 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-03-25 18:15:46.332701196 +0000 2024-03-25T19:34:35.753 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-03-25 18:15:46.332701196 +0000 2024-03-25T19:34:35.753 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-03-25 18:15:46.332701196 +0000 2024-03-25T19:34:35.754 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:34:35.818 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-03-25T19:34:35.819 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-03-25T19:34:35.819 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000196392 s, 2.6 MB/s 2024-03-25T19:34:35.820 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:34:35.876 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_4 2024-03-25T19:34:35.930 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:34:35.930 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:34:35.930 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-25T19:34:35.930 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:34:35.930 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:34:35.930 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-03-25 18:24:17.714116970 +0000 2024-03-25T19:34:35.930 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-03-25 18:15:46.661695384 +0000 2024-03-25T19:34:35.930 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-03-25 18:15:46.661695384 +0000 2024-03-25T19:34:35.930 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-03-25 18:15:46.661695384 +0000 2024-03-25T19:34:35.930 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:34:35.994 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-03-25T19:34:35.994 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-03-25T19:34:35.994 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000173856 s, 2.9 MB/s 2024-03-25T19:34:35.996 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:34:36.050 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:34:36.050 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi044.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi111.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:34:36.051 INFO:tasks.ceph:Generating config... 2024-03-25T19:34:36.052 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T19:34:36.052 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T19:34:36.052 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T19:34:36.052 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T19:34:36.052 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T19:34:36.052 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T19:34:36.052 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T19:34:36.052 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T19:34:36.052 INFO:tasks.ceph:[global] ms type = async 2024-03-25T19:34:36.052 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T19:34:36.052 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T19:34:36.052 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T19:34:36.052 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T19:34:36.052 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T19:34:36.052 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T19:34:36.053 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T19:34:36.053 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T19:34:36.053 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T19:34:36.053 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T19:34:36.053 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T19:34:36.053 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T19:34:36.053 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T19:34:36.053 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T19:34:36.053 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T19:34:36.053 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T19:34:36.053 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T19:34:36.053 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T19:34:36.053 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T19:34:36.053 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T19:34:36.053 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T19:34:36.053 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T19:34:36.053 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T19:34:36.054 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T19:34:36.054 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T19:34:36.054 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T19:34:36.054 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T19:34:36.054 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T19:34:36.054 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T19:34:36.054 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T19:34:36.054 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T19:34:36.054 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T19:34:36.054 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T19:34:36.054 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T19:34:36.054 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T19:34:36.054 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T19:34:36.054 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T19:34:36.054 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T19:34:36.055 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T19:34:36.055 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T19:34:36.055 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T19:34:36.055 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T19:34:36.055 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T19:34:36.055 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T19:34:36.055 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T19:34:36.055 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T19:34:36.113 INFO:teuthology.orchestra.run.smithi044.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T19:34:36.116 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T19:34:36.162 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:34:36.189 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.44:6789]'), ('mon.c', '[v1:172.21.15.44:6790]'), ('mon.b', '[v1:172.21.15.111:6789]')] 2024-03-25T19:34:36.190 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.44:6789],[v1:172.21.15.44:6790],[v1:172.21.15.111:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T19:34:36.190 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T19:34:36.190 DEBUG:teuthology.orchestra.run.smithi044:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:34:36.244 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.44:6789]' --addv c '[v1:172.21.15.44:6790]' --addv b '[v1:172.21.15.111:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:34:36.330 INFO:teuthology.orchestra.run.smithi044.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:34:36.330 INFO:teuthology.orchestra.run.smithi044.stdout:monmaptool: generated fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:34:36.330 INFO:teuthology.orchestra.run.smithi044.stdout:setting min_mon_release = quincy 2024-03-25T19:34:36.330 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 0 2024-03-25T19:34:36.330 INFO:teuthology.orchestra.run.smithi044.stdout:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:34:36.330 INFO:teuthology.orchestra.run.smithi044.stdout:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:34:36.330 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:34:36.330 INFO:teuthology.orchestra.run.smithi044.stdout:min_mon_release 17 (quincy) 2024-03-25T19:34:36.330 INFO:teuthology.orchestra.run.smithi044.stdout:election_strategy: 3 2024-03-25T19:34:36.330 INFO:teuthology.orchestra.run.smithi044.stdout:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:34:36.330 INFO:teuthology.orchestra.run.smithi044.stdout:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:34:36.330 INFO:teuthology.orchestra.run.smithi044.stdout:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:34:36.330 INFO:teuthology.orchestra.run.smithi044.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T19:34:36.331 DEBUG:teuthology.orchestra.run.smithi044:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:34:36.345 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f7c5aa74-cc2e-41df-a7e9-22cad1e45c88... 2024-03-25T19:34:36.346 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:34:36.388 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:34:36.435 INFO:teuthology.orchestra.run.smithi044.stdout:[global] 2024-03-25T19:34:36.435 INFO:teuthology.orchestra.run.smithi044.stdout: chdir = "" 2024-03-25T19:34:36.435 INFO:teuthology.orchestra.run.smithi044.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:34:36.435 INFO:teuthology.orchestra.run.smithi044.stdout: auth supported = cephx 2024-03-25T19:34:36.435 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.435 INFO:teuthology.orchestra.run.smithi044.stdout: filestore xattr use omap = true 2024-03-25T19:34:36.435 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.435 INFO:teuthology.orchestra.run.smithi044.stdout: mon clock drift allowed = 1.000 2024-03-25T19:34:36.435 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.435 INFO:teuthology.orchestra.run.smithi044.stdout: osd crush chooseleaf type = 0 2024-03-25T19:34:36.435 INFO:teuthology.orchestra.run.smithi044.stdout: auth debug = true 2024-03-25T19:34:36.436 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.436 INFO:teuthology.orchestra.run.smithi044.stdout: ms die on old message = true 2024-03-25T19:34:36.436 INFO:teuthology.orchestra.run.smithi044.stdout: ms die on bug = true 2024-03-25T19:34:36.436 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.436 INFO:teuthology.orchestra.run.smithi044.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:34:36.436 INFO:teuthology.orchestra.run.smithi044.stdout: mon pg warn max object skew = 0 2024-03-25T19:34:36.436 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.436 INFO:teuthology.orchestra.run.smithi044.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:34:36.436 INFO:teuthology.orchestra.run.smithi044.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:34:36.436 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.436 INFO:teuthology.orchestra.run.smithi044.stdout: osd pool default size = 2 2024-03-25T19:34:36.436 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.436 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd allow primary affinity = true 2024-03-25T19:34:36.436 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd allow pg remap = true 2024-03-25T19:34:36.436 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:34:36.437 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:34:36.437 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on no sortbitwise = false 2024-03-25T19:34:36.437 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:34:36.437 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on too few osds = false 2024-03-25T19:34:36.437 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:34:36.437 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:34:36.437 INFO:teuthology.orchestra.run.smithi044.stdout: mon_allow_pool_size_one = true 2024-03-25T19:34:36.437 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.437 INFO:teuthology.orchestra.run.smithi044.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:34:36.437 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.437 INFO:teuthology.orchestra.run.smithi044.stdout: osd default data pool replay window = 5 2024-03-25T19:34:36.437 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.437 INFO:teuthology.orchestra.run.smithi044.stdout: mon allow pool delete = true 2024-03-25T19:34:36.437 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.437 INFO:teuthology.orchestra.run.smithi044.stdout: mon cluster log file level = debug 2024-03-25T19:34:36.437 INFO:teuthology.orchestra.run.smithi044.stdout: debug asserts on shutdown = true 2024-03-25T19:34:36.438 INFO:teuthology.orchestra.run.smithi044.stdout: mon health detail to clog = false 2024-03-25T19:34:36.438 INFO:teuthology.orchestra.run.smithi044.stdout: mon host = "[v1:172.21.15.44:6789],[v1:172.21.15.44:6790],[v1:172.21.15.111:6789]" 2024-03-25T19:34:36.438 INFO:teuthology.orchestra.run.smithi044.stdout: mon client directed command retry = 5 2024-03-25T19:34:36.438 INFO:teuthology.orchestra.run.smithi044.stdout: mon election default strategy = 3 2024-03-25T19:34:36.438 INFO:teuthology.orchestra.run.smithi044.stdout: ms bind msgr2 = False 2024-03-25T19:34:36.438 INFO:teuthology.orchestra.run.smithi044.stdout: ms inject socket failures = 5000 2024-03-25T19:34:36.438 INFO:teuthology.orchestra.run.smithi044.stdout: ms type = async 2024-03-25T19:34:36.438 INFO:teuthology.orchestra.run.smithi044.stdout: osd heartbeat grace = 80 2024-03-25T19:34:36.438 INFO:teuthology.orchestra.run.smithi044.stdout: fsid = f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:34:36.438 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.438 INFO:teuthology.orchestra.run.smithi044.stdout:[osd] 2024-03-25T19:34:36.438 INFO:teuthology.orchestra.run.smithi044.stdout: osd journal size = 100 2024-03-25T19:34:36.438 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.438 INFO:teuthology.orchestra.run.smithi044.stdout: osd scrub load threshold = 5.0 2024-03-25T19:34:36.439 INFO:teuthology.orchestra.run.smithi044.stdout: osd scrub max interval = 600 2024-03-25T19:34:36.439 INFO:teuthology.orchestra.run.smithi044.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:34:36.439 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.439 INFO:teuthology.orchestra.run.smithi044.stdout: osd recover clone overlap = true 2024-03-25T19:34:36.439 INFO:teuthology.orchestra.run.smithi044.stdout: osd recovery max chunk = 1048576 2024-03-25T19:34:36.439 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.439 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug shutdown = true 2024-03-25T19:34:36.439 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug op order = true 2024-03-25T19:34:36.439 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug verify stray on activate = true 2024-03-25T19:34:36.439 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug trim objects = true 2024-03-25T19:34:36.439 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.439 INFO:teuthology.orchestra.run.smithi044.stdout: osd open classes on start = true 2024-03-25T19:34:36.439 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug pg log writeout = true 2024-03-25T19:34:36.439 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.439 INFO:teuthology.orchestra.run.smithi044.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:34:36.440 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.440 INFO:teuthology.orchestra.run.smithi044.stdout: osd map max advance = 10 2024-03-25T19:34:36.440 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.440 INFO:teuthology.orchestra.run.smithi044.stdout: journal zero on create = true 2024-03-25T19:34:36.440 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.440 INFO:teuthology.orchestra.run.smithi044.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:34:36.440 INFO:teuthology.orchestra.run.smithi044.stdout: filestore apply finisher threads = 3 2024-03-25T19:34:36.440 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.440 INFO:teuthology.orchestra.run.smithi044.stdout: bdev debug aio = true 2024-03-25T19:34:36.440 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug misdirected ops = true 2024-03-25T19:34:36.440 INFO:teuthology.orchestra.run.smithi044.stdout: bdev async discard = True 2024-03-25T19:34:36.440 INFO:teuthology.orchestra.run.smithi044.stdout: bdev enable discard = True 2024-03-25T19:34:36.440 INFO:teuthology.orchestra.run.smithi044.stdout: bluestore allocator = stupid 2024-03-25T19:34:36.440 INFO:teuthology.orchestra.run.smithi044.stdout: bluestore block size = 96636764160 2024-03-25T19:34:36.440 INFO:teuthology.orchestra.run.smithi044.stdout: bluestore fsck on mount = True 2024-03-25T19:34:36.440 INFO:teuthology.orchestra.run.smithi044.stdout: bluestore zero block detection = True 2024-03-25T19:34:36.441 INFO:teuthology.orchestra.run.smithi044.stdout: debug bluefs = 1 2024-03-25T19:34:36.441 INFO:teuthology.orchestra.run.smithi044.stdout: debug bluestore = 1 2024-03-25T19:34:36.441 INFO:teuthology.orchestra.run.smithi044.stdout: debug monc = 20 2024-03-25T19:34:36.441 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-03-25T19:34:36.441 INFO:teuthology.orchestra.run.smithi044.stdout: debug osd = 20 2024-03-25T19:34:36.441 INFO:teuthology.orchestra.run.smithi044.stdout: debug rocksdb = 10 2024-03-25T19:34:36.441 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:34:36.441 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd full ratio = 0.9 2024-03-25T19:34:36.441 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:34:36.441 INFO:teuthology.orchestra.run.smithi044.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:34:36.441 INFO:teuthology.orchestra.run.smithi044.stdout: osd class default list = * 2024-03-25T19:34:36.441 INFO:teuthology.orchestra.run.smithi044.stdout: osd class load list = * 2024-03-25T19:34:36.441 INFO:teuthology.orchestra.run.smithi044.stdout: osd client watch timeout = 120 2024-03-25T19:34:36.441 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug verify cached snaps = True 2024-03-25T19:34:36.441 INFO:teuthology.orchestra.run.smithi044.stdout: osd debug verify missing on start = True 2024-03-25T19:34:36.441 INFO:teuthology.orchestra.run.smithi044.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:34:36.442 INFO:teuthology.orchestra.run.smithi044.stdout: osd fast shutdown = False 2024-03-25T19:34:36.442 INFO:teuthology.orchestra.run.smithi044.stdout: osd max markdown count = 1000 2024-03-25T19:34:36.442 INFO:teuthology.orchestra.run.smithi044.stdout: osd mclock override recovery settings = True 2024-03-25T19:34:36.442 INFO:teuthology.orchestra.run.smithi044.stdout: osd objectstore = bluestore 2024-03-25T19:34:36.442 INFO:teuthology.orchestra.run.smithi044.stdout: osd op queue = debug_random 2024-03-25T19:34:36.442 INFO:teuthology.orchestra.run.smithi044.stdout: osd op queue cut off = debug_random 2024-03-25T19:34:36.442 INFO:teuthology.orchestra.run.smithi044.stdout: osd shutdown pgref assert = True 2024-03-25T19:34:36.442 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.442 INFO:teuthology.orchestra.run.smithi044.stdout:[mgr] 2024-03-25T19:34:36.442 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-03-25T19:34:36.442 INFO:teuthology.orchestra.run.smithi044.stdout: debug mgr = 20 2024-03-25T19:34:36.442 INFO:teuthology.orchestra.run.smithi044.stdout: debug mon = 20 2024-03-25T19:34:36.442 INFO:teuthology.orchestra.run.smithi044.stdout: debug auth = 20 2024-03-25T19:34:36.442 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:34:36.443 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:34:36.443 INFO:teuthology.orchestra.run.smithi044.stdout: mgr/telemetry/nag = false 2024-03-25T19:34:36.443 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.443 INFO:teuthology.orchestra.run.smithi044.stdout:[mon] 2024-03-25T19:34:36.443 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-03-25T19:34:36.443 INFO:teuthology.orchestra.run.smithi044.stdout: debug mon = 20 2024-03-25T19:34:36.443 INFO:teuthology.orchestra.run.smithi044.stdout: debug paxos = 20 2024-03-25T19:34:36.443 INFO:teuthology.orchestra.run.smithi044.stdout: debug auth = 20 2024-03-25T19:34:36.443 INFO:teuthology.orchestra.run.smithi044.stdout: mon data avail warn = 5 2024-03-25T19:34:36.443 INFO:teuthology.orchestra.run.smithi044.stdout: mon mgr mkfs grace = 240 2024-03-25T19:34:36.443 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:34:36.443 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd reporter subtree level = osd 2024-03-25T19:34:36.443 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd prime pg temp = true 2024-03-25T19:34:36.443 INFO:teuthology.orchestra.run.smithi044.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:34:36.443 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.443 INFO:teuthology.orchestra.run.smithi044.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:34:36.443 INFO:teuthology.orchestra.run.smithi044.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:34:36.444 INFO:teuthology.orchestra.run.smithi044.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:34:36.444 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.444 INFO:teuthology.orchestra.run.smithi044.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:34:36.444 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:34:36.444 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:34:36.444 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.444 INFO:teuthology.orchestra.run.smithi044.stdout: # 1m isn't quite enough 2024-03-25T19:34:36.444 INFO:teuthology.orchestra.run.smithi044.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:34:36.444 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.444 INFO:teuthology.orchestra.run.smithi044.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:34:36.444 INFO:teuthology.orchestra.run.smithi044.stdout: mon min osdmap epochs = 50 2024-03-25T19:34:36.444 INFO:teuthology.orchestra.run.smithi044.stdout: mon osd crush smoke test = False 2024-03-25T19:34:36.444 INFO:teuthology.orchestra.run.smithi044.stdout: mon osdmap full prune interval = 2 2024-03-25T19:34:36.444 INFO:teuthology.orchestra.run.smithi044.stdout: mon osdmap full prune min = 15 2024-03-25T19:34:36.444 INFO:teuthology.orchestra.run.smithi044.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:34:36.444 INFO:teuthology.orchestra.run.smithi044.stdout: mon scrub interval = 300 2024-03-25T19:34:36.445 INFO:teuthology.orchestra.run.smithi044.stdout: mon warn on pool no app = False 2024-03-25T19:34:36.445 INFO:teuthology.orchestra.run.smithi044.stdout: paxos service trim min = 10 2024-03-25T19:34:36.445 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:34:36.445 INFO:teuthology.orchestra.run.smithi044.stdout:[client] 2024-03-25T19:34:36.445 INFO:teuthology.orchestra.run.smithi044.stdout: rgw cache enabled = true 2024-03-25T19:34:36.445 INFO:teuthology.orchestra.run.smithi044.stdout: rgw enable ops log = true 2024-03-25T19:34:36.445 INFO:teuthology.orchestra.run.smithi044.stdout: rgw enable usage log = true 2024-03-25T19:34:36.445 INFO:teuthology.orchestra.run.smithi044.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:34:36.445 INFO:teuthology.orchestra.run.smithi044.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:34:36.445 INFO:teuthology.orchestra.run.smithi044.stdout: debug monc = 20 2024-03-25T19:34:36.445 INFO:teuthology.orchestra.run.smithi044.stdout: debug ms = 1 2024-03-25T19:34:36.445 INFO:teuthology.orchestra.run.smithi044.stdout: debug objecter = 20 2024-03-25T19:34:36.445 INFO:teuthology.orchestra.run.smithi044.stdout: debug rados = 20 2024-03-25T19:34:36.445 INFO:teuthology.orchestra.run.smithi044.stdout:[mon.a] 2024-03-25T19:34:36.445 INFO:teuthology.orchestra.run.smithi044.stdout:[mon.c] 2024-03-25T19:34:36.445 INFO:teuthology.orchestra.run.smithi044.stdout:[mon.b] 2024-03-25T19:34:36.446 INFO:teuthology.orchestra.run.smithi111.stdout:[global] 2024-03-25T19:34:36.446 INFO:teuthology.orchestra.run.smithi111.stdout: chdir = "" 2024-03-25T19:34:36.446 INFO:teuthology.orchestra.run.smithi111.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:34:36.446 INFO:teuthology.orchestra.run.smithi111.stdout: auth supported = cephx 2024-03-25T19:34:36.446 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.446 INFO:teuthology.orchestra.run.smithi111.stdout: filestore xattr use omap = true 2024-03-25T19:34:36.446 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.446 INFO:teuthology.orchestra.run.smithi111.stdout: mon clock drift allowed = 1.000 2024-03-25T19:34:36.446 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.447 INFO:teuthology.orchestra.run.smithi111.stdout: osd crush chooseleaf type = 0 2024-03-25T19:34:36.447 INFO:teuthology.orchestra.run.smithi111.stdout: auth debug = true 2024-03-25T19:34:36.447 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.447 INFO:teuthology.orchestra.run.smithi111.stdout: ms die on old message = true 2024-03-25T19:34:36.447 INFO:teuthology.orchestra.run.smithi111.stdout: ms die on bug = true 2024-03-25T19:34:36.447 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.447 INFO:teuthology.orchestra.run.smithi111.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:34:36.447 INFO:teuthology.orchestra.run.smithi111.stdout: mon pg warn max object skew = 0 2024-03-25T19:34:36.447 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.447 INFO:teuthology.orchestra.run.smithi111.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:34:36.447 INFO:teuthology.orchestra.run.smithi111.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:34:36.447 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.447 INFO:teuthology.orchestra.run.smithi111.stdout: osd pool default size = 2 2024-03-25T19:34:36.447 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.447 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd allow primary affinity = true 2024-03-25T19:34:36.447 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd allow pg remap = true 2024-03-25T19:34:36.448 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:34:36.448 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:34:36.448 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on no sortbitwise = false 2024-03-25T19:34:36.448 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:34:36.448 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on too few osds = false 2024-03-25T19:34:36.448 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:34:36.448 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:34:36.448 INFO:teuthology.orchestra.run.smithi111.stdout: mon_allow_pool_size_one = true 2024-03-25T19:34:36.448 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.448 INFO:teuthology.orchestra.run.smithi111.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:34:36.448 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.448 INFO:teuthology.orchestra.run.smithi111.stdout: osd default data pool replay window = 5 2024-03-25T19:34:36.448 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.448 INFO:teuthology.orchestra.run.smithi111.stdout: mon allow pool delete = true 2024-03-25T19:34:36.448 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.448 INFO:teuthology.orchestra.run.smithi111.stdout: mon cluster log file level = debug 2024-03-25T19:34:36.449 INFO:teuthology.orchestra.run.smithi111.stdout: debug asserts on shutdown = true 2024-03-25T19:34:36.449 INFO:teuthology.orchestra.run.smithi111.stdout: mon health detail to clog = false 2024-03-25T19:34:36.449 INFO:teuthology.orchestra.run.smithi111.stdout: mon host = "[v1:172.21.15.44:6789],[v1:172.21.15.44:6790],[v1:172.21.15.111:6789]" 2024-03-25T19:34:36.449 INFO:teuthology.orchestra.run.smithi111.stdout: mon client directed command retry = 5 2024-03-25T19:34:36.449 INFO:teuthology.orchestra.run.smithi111.stdout: mon election default strategy = 3 2024-03-25T19:34:36.449 INFO:teuthology.orchestra.run.smithi111.stdout: ms bind msgr2 = False 2024-03-25T19:34:36.449 INFO:teuthology.orchestra.run.smithi111.stdout: ms inject socket failures = 5000 2024-03-25T19:34:36.449 INFO:teuthology.orchestra.run.smithi111.stdout: ms type = async 2024-03-25T19:34:36.449 INFO:teuthology.orchestra.run.smithi111.stdout: osd heartbeat grace = 80 2024-03-25T19:34:36.449 INFO:teuthology.orchestra.run.smithi111.stdout: fsid = f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:34:36.449 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.449 INFO:teuthology.orchestra.run.smithi111.stdout:[osd] 2024-03-25T19:34:36.449 INFO:teuthology.orchestra.run.smithi111.stdout: osd journal size = 100 2024-03-25T19:34:36.449 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.449 INFO:teuthology.orchestra.run.smithi111.stdout: osd scrub load threshold = 5.0 2024-03-25T19:34:36.449 INFO:teuthology.orchestra.run.smithi111.stdout: osd scrub max interval = 600 2024-03-25T19:34:36.450 INFO:teuthology.orchestra.run.smithi111.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:34:36.450 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.450 INFO:teuthology.orchestra.run.smithi111.stdout: osd recover clone overlap = true 2024-03-25T19:34:36.450 INFO:teuthology.orchestra.run.smithi111.stdout: osd recovery max chunk = 1048576 2024-03-25T19:34:36.450 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.450 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug shutdown = true 2024-03-25T19:34:36.450 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug op order = true 2024-03-25T19:34:36.450 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug verify stray on activate = true 2024-03-25T19:34:36.450 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug trim objects = true 2024-03-25T19:34:36.450 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.450 INFO:teuthology.orchestra.run.smithi111.stdout: osd open classes on start = true 2024-03-25T19:34:36.450 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug pg log writeout = true 2024-03-25T19:34:36.450 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.450 INFO:teuthology.orchestra.run.smithi111.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:34:36.450 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.451 INFO:teuthology.orchestra.run.smithi111.stdout: osd map max advance = 10 2024-03-25T19:34:36.451 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.451 INFO:teuthology.orchestra.run.smithi111.stdout: journal zero on create = true 2024-03-25T19:34:36.451 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.451 INFO:teuthology.orchestra.run.smithi111.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:34:36.451 INFO:teuthology.orchestra.run.smithi111.stdout: filestore apply finisher threads = 3 2024-03-25T19:34:36.451 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.451 INFO:teuthology.orchestra.run.smithi111.stdout: bdev debug aio = true 2024-03-25T19:34:36.451 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug misdirected ops = true 2024-03-25T19:34:36.451 INFO:teuthology.orchestra.run.smithi111.stdout: bdev async discard = True 2024-03-25T19:34:36.451 INFO:teuthology.orchestra.run.smithi111.stdout: bdev enable discard = True 2024-03-25T19:34:36.451 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore allocator = stupid 2024-03-25T19:34:36.451 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore block size = 96636764160 2024-03-25T19:34:36.451 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore fsck on mount = True 2024-03-25T19:34:36.451 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore zero block detection = True 2024-03-25T19:34:36.451 INFO:teuthology.orchestra.run.smithi111.stdout: debug bluefs = 1 2024-03-25T19:34:36.451 INFO:teuthology.orchestra.run.smithi111.stdout: debug bluestore = 1 2024-03-25T19:34:36.452 INFO:teuthology.orchestra.run.smithi111.stdout: debug monc = 20 2024-03-25T19:34:36.452 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2024-03-25T19:34:36.452 INFO:teuthology.orchestra.run.smithi111.stdout: debug osd = 20 2024-03-25T19:34:36.452 INFO:teuthology.orchestra.run.smithi111.stdout: debug rocksdb = 10 2024-03-25T19:34:36.452 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:34:36.452 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd full ratio = 0.9 2024-03-25T19:34:36.452 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:34:36.452 INFO:teuthology.orchestra.run.smithi111.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:34:36.452 INFO:teuthology.orchestra.run.smithi111.stdout: osd class default list = * 2024-03-25T19:34:36.452 INFO:teuthology.orchestra.run.smithi111.stdout: osd class load list = * 2024-03-25T19:34:36.452 INFO:teuthology.orchestra.run.smithi111.stdout: osd client watch timeout = 120 2024-03-25T19:34:36.452 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug verify cached snaps = True 2024-03-25T19:34:36.452 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug verify missing on start = True 2024-03-25T19:34:36.452 INFO:teuthology.orchestra.run.smithi111.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:34:36.452 INFO:teuthology.orchestra.run.smithi111.stdout: osd fast shutdown = False 2024-03-25T19:34:36.452 INFO:teuthology.orchestra.run.smithi111.stdout: osd max markdown count = 1000 2024-03-25T19:34:36.453 INFO:teuthology.orchestra.run.smithi111.stdout: osd mclock override recovery settings = True 2024-03-25T19:34:36.453 INFO:teuthology.orchestra.run.smithi111.stdout: osd objectstore = bluestore 2024-03-25T19:34:36.453 INFO:teuthology.orchestra.run.smithi111.stdout: osd op queue = debug_random 2024-03-25T19:34:36.453 INFO:teuthology.orchestra.run.smithi111.stdout: osd op queue cut off = debug_random 2024-03-25T19:34:36.453 INFO:teuthology.orchestra.run.smithi111.stdout: osd shutdown pgref assert = True 2024-03-25T19:34:36.453 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.453 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr] 2024-03-25T19:34:36.453 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2024-03-25T19:34:36.453 INFO:teuthology.orchestra.run.smithi111.stdout: debug mgr = 20 2024-03-25T19:34:36.453 INFO:teuthology.orchestra.run.smithi111.stdout: debug mon = 20 2024-03-25T19:34:36.453 INFO:teuthology.orchestra.run.smithi111.stdout: debug auth = 20 2024-03-25T19:34:36.453 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:34:36.453 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:34:36.453 INFO:teuthology.orchestra.run.smithi111.stdout: mgr/telemetry/nag = false 2024-03-25T19:34:36.453 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.453 INFO:teuthology.orchestra.run.smithi111.stdout:[mon] 2024-03-25T19:34:36.454 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2024-03-25T19:34:36.454 INFO:teuthology.orchestra.run.smithi111.stdout: debug mon = 20 2024-03-25T19:34:36.454 INFO:teuthology.orchestra.run.smithi111.stdout: debug paxos = 20 2024-03-25T19:34:36.454 INFO:teuthology.orchestra.run.smithi111.stdout: debug auth = 20 2024-03-25T19:34:36.454 INFO:teuthology.orchestra.run.smithi111.stdout: mon data avail warn = 5 2024-03-25T19:34:36.454 INFO:teuthology.orchestra.run.smithi111.stdout: mon mgr mkfs grace = 240 2024-03-25T19:34:36.454 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:34:36.454 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd reporter subtree level = osd 2024-03-25T19:34:36.454 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd prime pg temp = true 2024-03-25T19:34:36.454 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:34:36.454 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.454 INFO:teuthology.orchestra.run.smithi111.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:34:36.454 INFO:teuthology.orchestra.run.smithi111.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:34:36.454 INFO:teuthology.orchestra.run.smithi111.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:34:36.455 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.455 INFO:teuthology.orchestra.run.smithi111.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:34:36.455 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:34:36.455 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:34:36.455 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.455 INFO:teuthology.orchestra.run.smithi111.stdout: # 1m isn't quite enough 2024-03-25T19:34:36.455 INFO:teuthology.orchestra.run.smithi111.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:34:36.455 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.455 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:34:36.455 INFO:teuthology.orchestra.run.smithi111.stdout: mon min osdmap epochs = 50 2024-03-25T19:34:36.455 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd crush smoke test = False 2024-03-25T19:34:36.455 INFO:teuthology.orchestra.run.smithi111.stdout: mon osdmap full prune interval = 2 2024-03-25T19:34:36.455 INFO:teuthology.orchestra.run.smithi111.stdout: mon osdmap full prune min = 15 2024-03-25T19:34:36.455 INFO:teuthology.orchestra.run.smithi111.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:34:36.455 INFO:teuthology.orchestra.run.smithi111.stdout: mon scrub interval = 300 2024-03-25T19:34:36.456 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on pool no app = False 2024-03-25T19:34:36.456 INFO:teuthology.orchestra.run.smithi111.stdout: paxos service trim min = 10 2024-03-25T19:34:36.456 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-25T19:34:36.456 INFO:teuthology.orchestra.run.smithi111.stdout:[client] 2024-03-25T19:34:36.456 INFO:teuthology.orchestra.run.smithi111.stdout: rgw cache enabled = true 2024-03-25T19:34:36.456 INFO:teuthology.orchestra.run.smithi111.stdout: rgw enable ops log = true 2024-03-25T19:34:36.456 INFO:teuthology.orchestra.run.smithi111.stdout: rgw enable usage log = true 2024-03-25T19:34:36.456 INFO:teuthology.orchestra.run.smithi111.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:34:36.456 INFO:teuthology.orchestra.run.smithi111.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:34:36.456 INFO:teuthology.orchestra.run.smithi111.stdout: debug monc = 20 2024-03-25T19:34:36.456 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2024-03-25T19:34:36.456 INFO:teuthology.orchestra.run.smithi111.stdout: debug objecter = 20 2024-03-25T19:34:36.456 INFO:teuthology.orchestra.run.smithi111.stdout: debug rados = 20 2024-03-25T19:34:36.456 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.a] 2024-03-25T19:34:36.456 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.c] 2024-03-25T19:34:36.457 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.b] 2024-03-25T19:34:36.458 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T19:34:36.458 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T19:34:36.512 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T19:34:36.512 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T19:34:36.512 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T19:34:36.526 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T19:34:36.526 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T19:34:36.581 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi044.front.sepia.ceph.com 2024-03-25T19:34:36.581 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T19:34:36.581 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:34:36.581 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:34:36.659 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T19:34:36.660 DEBUG:teuthology.orchestra.run.smithi044:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:34:36.714 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi111.front.sepia.ceph.com 2024-03-25T19:34:36.714 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T19:34:36.714 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:34:36.714 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:34:36.752 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T19:34:36.752 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:34:36.807 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T19:34:36.807 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T19:34:36.807 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:34:36.862 INFO:teuthology.orchestra.run.smithi044.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:34:36.865 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:34:36.928 INFO:teuthology.orchestra.run.smithi111.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:34:36.931 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T19:34:36.931 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T19:34:36.931 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T19:34:36.973 INFO:teuthology.orchestra.run.smithi044.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T19:34:36.988 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T19:34:37.030 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T19:34:37.045 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T19:34:37.046 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi044.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi111.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:34:37.046 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T19:34:37.072 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:34:37.072 INFO:tasks.ceph:role: osd.0 2024-03-25T19:34:37.072 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi044.front.sepia.ceph.com 2024-03-25T19:34:37.072 DEBUG:teuthology.orchestra.run.smithi044:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:34:37.146 INFO:teuthology.orchestra.run.smithi044.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:34:37.146 INFO:teuthology.orchestra.run.smithi044.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:34:37.147 INFO:teuthology.orchestra.run.smithi044.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:34:37.147 INFO:teuthology.orchestra.run.smithi044.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:34:37.147 INFO:teuthology.orchestra.run.smithi044.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:34:37.147 INFO:teuthology.orchestra.run.smithi044.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:34:37.147 INFO:teuthology.orchestra.run.smithi044.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:34:37.147 INFO:teuthology.orchestra.run.smithi044.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:34:37.147 INFO:teuthology.orchestra.run.smithi044.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:34:37.147 INFO:teuthology.orchestra.run.smithi044.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:34:37.650 INFO:teuthology.orchestra.run.smithi044.stdout:Discarding blocks...Done. 2024-03-25T19:34:37.652 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi044.front.sepia.ceph.com -o noatime 2024-03-25T19:34:37.652 DEBUG:teuthology.orchestra.run.smithi044:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T19:34:37.754 DEBUG:teuthology.orchestra.run.smithi044:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T19:34:37.788 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T19:34:37.855 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:34:37.855 INFO:tasks.ceph:role: osd.1 2024-03-25T19:34:37.856 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi044.front.sepia.ceph.com 2024-03-25T19:34:37.856 DEBUG:teuthology.orchestra.run.smithi044:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:34:37.929 INFO:teuthology.orchestra.run.smithi044.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:34:37.929 INFO:teuthology.orchestra.run.smithi044.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:34:37.930 INFO:teuthology.orchestra.run.smithi044.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:34:37.930 INFO:teuthology.orchestra.run.smithi044.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:34:37.930 INFO:teuthology.orchestra.run.smithi044.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:34:37.930 INFO:teuthology.orchestra.run.smithi044.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:34:37.930 INFO:teuthology.orchestra.run.smithi044.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:34:37.930 INFO:teuthology.orchestra.run.smithi044.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:34:37.930 INFO:teuthology.orchestra.run.smithi044.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:34:37.930 INFO:teuthology.orchestra.run.smithi044.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:34:38.385 INFO:teuthology.orchestra.run.smithi044.stdout:Discarding blocks...Done. 2024-03-25T19:34:38.387 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi044.front.sepia.ceph.com -o noatime 2024-03-25T19:34:38.387 DEBUG:teuthology.orchestra.run.smithi044:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T19:34:38.479 DEBUG:teuthology.orchestra.run.smithi044:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T19:34:38.510 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T19:34:38.577 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:34:38.577 INFO:tasks.ceph:role: osd.2 2024-03-25T19:34:38.577 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi044.front.sepia.ceph.com 2024-03-25T19:34:38.577 DEBUG:teuthology.orchestra.run.smithi044:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:34:38.651 INFO:teuthology.orchestra.run.smithi044.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:34:38.651 INFO:teuthology.orchestra.run.smithi044.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:34:38.651 INFO:teuthology.orchestra.run.smithi044.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:34:38.651 INFO:teuthology.orchestra.run.smithi044.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:34:38.652 INFO:teuthology.orchestra.run.smithi044.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:34:38.652 INFO:teuthology.orchestra.run.smithi044.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:34:38.652 INFO:teuthology.orchestra.run.smithi044.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:34:38.652 INFO:teuthology.orchestra.run.smithi044.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:34:38.652 INFO:teuthology.orchestra.run.smithi044.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:34:38.652 INFO:teuthology.orchestra.run.smithi044.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:34:39.129 INFO:teuthology.orchestra.run.smithi044.stdout:Discarding blocks...Done. 2024-03-25T19:34:39.131 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi044.front.sepia.ceph.com -o noatime 2024-03-25T19:34:39.131 DEBUG:teuthology.orchestra.run.smithi044:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T19:34:39.274 DEBUG:teuthology.orchestra.run.smithi044:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T19:34:39.307 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T19:34:39.374 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:34:39.374 INFO:tasks.ceph:role: osd.3 2024-03-25T19:34:39.374 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi044.front.sepia.ceph.com 2024-03-25T19:34:39.374 DEBUG:teuthology.orchestra.run.smithi044:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:34:39.449 INFO:teuthology.orchestra.run.smithi044.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:34:39.449 INFO:teuthology.orchestra.run.smithi044.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:34:39.450 INFO:teuthology.orchestra.run.smithi044.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:34:39.450 INFO:teuthology.orchestra.run.smithi044.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:34:39.450 INFO:teuthology.orchestra.run.smithi044.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:34:39.450 INFO:teuthology.orchestra.run.smithi044.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:34:39.450 INFO:teuthology.orchestra.run.smithi044.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:34:39.450 INFO:teuthology.orchestra.run.smithi044.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:34:39.450 INFO:teuthology.orchestra.run.smithi044.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:34:39.450 INFO:teuthology.orchestra.run.smithi044.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:34:39.935 INFO:teuthology.orchestra.run.smithi044.stdout:Discarding blocks...Done. 2024-03-25T19:34:39.937 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi044.front.sepia.ceph.com -o noatime 2024-03-25T19:34:39.937 DEBUG:teuthology.orchestra.run.smithi044:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T19:34:40.077 DEBUG:teuthology.orchestra.run.smithi044:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T19:34:40.112 DEBUG:teuthology.orchestra.run.smithi044:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:34:40.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:34:40.214+0000 7f17a1414640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T19:34:40.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:34:40.214+0000 7f17a1414640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T19:34:40.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:34:40.214+0000 7f17a1414640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:34:40.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:34:40.214+0000 7f17a1414640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:34:40.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:34:40.214+0000 7f17a1414640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:34:40.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:34:40.215+0000 7f17a1414640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T19:34:43.066 DEBUG:teuthology.orchestra.run.smithi044:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T19:34:43.095 DEBUG:teuthology.orchestra.run.smithi044:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:34:43.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:34:43.183+0000 7f5696c0e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T19:34:43.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:34:43.184+0000 7f5696c0e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T19:34:43.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:34:43.184+0000 7f5696c0e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:34:43.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:34:43.184+0000 7f5696c0e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:34:43.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:34:43.184+0000 7f5696c0e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:34:43.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:34:43.184+0000 7f5696c0e640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T19:34:46.022 DEBUG:teuthology.orchestra.run.smithi044:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T19:34:46.053 DEBUG:teuthology.orchestra.run.smithi044:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:34:46.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:34:46.144+0000 7fd732a15640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T19:34:46.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:34:46.144+0000 7fd732a15640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T19:34:46.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:34:46.144+0000 7fd732a15640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:34:46.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:34:46.144+0000 7fd732a15640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:34:46.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:34:46.144+0000 7fd732a15640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:34:46.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:34:46.145+0000 7fd732a15640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T19:34:48.995 DEBUG:teuthology.orchestra.run.smithi044:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T19:34:49.025 DEBUG:teuthology.orchestra.run.smithi044:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:34:49.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:34:49.117+0000 7f6719015640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T19:34:49.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:34:49.117+0000 7f6719015640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T19:34:49.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:34:49.117+0000 7f6719015640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:34:49.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:34:49.117+0000 7f6719015640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:34:49.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:34:49.117+0000 7f6719015640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:34:49.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:34:49.118+0000 7f6719015640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T19:34:51.957 DEBUG:teuthology.orchestra.run.smithi044:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T19:34:51.987 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T19:34:52.019 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:34:52.019 INFO:tasks.ceph:role: osd.4 2024-03-25T19:34:52.019 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi111.front.sepia.ceph.com 2024-03-25T19:34:52.019 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:34:52.097 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:34:52.098 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:34:52.098 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:34:52.098 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:34:52.098 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:34:52.098 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:34:52.098 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:34:52.098 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:34:52.098 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:34:52.098 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:34:52.604 INFO:teuthology.orchestra.run.smithi111.stdout:Discarding blocks...Done. 2024-03-25T19:34:52.606 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2024-03-25T19:34:52.606 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T19:34:52.734 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T19:34:52.771 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T19:34:52.841 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:34:52.841 INFO:tasks.ceph:role: osd.5 2024-03-25T19:34:52.841 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi111.front.sepia.ceph.com 2024-03-25T19:34:52.841 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:34:52.917 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:34:52.917 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:34:52.917 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:34:52.917 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:34:52.917 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:34:52.918 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:34:52.918 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:34:52.918 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:34:52.918 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:34:52.918 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:34:53.373 INFO:teuthology.orchestra.run.smithi111.stdout:Discarding blocks...Done. 2024-03-25T19:34:53.376 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2024-03-25T19:34:53.376 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T19:34:53.469 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T19:34:53.503 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T19:34:53.572 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:34:53.572 INFO:tasks.ceph:role: osd.6 2024-03-25T19:34:53.572 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi111.front.sepia.ceph.com 2024-03-25T19:34:53.572 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:34:53.648 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:34:53.648 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:34:53.649 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:34:53.649 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:34:53.649 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:34:53.649 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:34:53.649 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:34:53.649 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:34:53.649 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:34:53.649 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:34:54.130 INFO:teuthology.orchestra.run.smithi111.stdout:Discarding blocks...Done. 2024-03-25T19:34:54.132 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2024-03-25T19:34:54.132 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T19:34:54.255 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T19:34:54.289 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T19:34:54.358 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:34:54.358 INFO:tasks.ceph:role: osd.7 2024-03-25T19:34:54.358 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi111.front.sepia.ceph.com 2024-03-25T19:34:54.358 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:34:54.434 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:34:54.434 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:34:54.434 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:34:54.435 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:34:54.435 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:34:54.435 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:34:54.435 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:34:54.435 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:34:54.435 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:34:54.435 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:34:54.932 INFO:teuthology.orchestra.run.smithi111.stdout:Discarding blocks...Done. 2024-03-25T19:34:54.934 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2024-03-25T19:34:54.934 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T19:34:55.064 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T19:34:55.100 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:34:55.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:34:55.205+0000 7f3a91c13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T19:34:55.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:34:55.205+0000 7f3a91c13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T19:34:55.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:34:55.206+0000 7f3a91c13640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:34:55.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:34:55.206+0000 7f3a91c13640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:34:55.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:34:55.206+0000 7f3a91c13640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:34:55.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:34:55.206+0000 7f3a91c13640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T19:34:58.046 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T19:34:58.074 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:34:58.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:34:58.161+0000 7f154a71f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T19:34:58.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:34:58.161+0000 7f154a71f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T19:34:58.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:34:58.161+0000 7f154a71f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:34:58.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:34:58.161+0000 7f154a71f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:34:58.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:34:58.161+0000 7f154a71f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:34:58.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:34:58.162+0000 7f154a71f640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T19:35:01.002 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T19:35:01.032 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:35:01.123 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:01.123+0000 7f0f7c538640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T19:35:01.124 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:01.124+0000 7f0f7c538640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T19:35:01.124 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:01.124+0000 7f0f7c538640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:35:01.124 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:01.124+0000 7f0f7c538640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:35:01.124 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:01.124+0000 7f0f7c538640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:35:01.124 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:01.124+0000 7f0f7c538640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T19:35:03.976 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T19:35:04.005 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:35:04.094 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:04.094+0000 7f2b838f9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T19:35:04.094 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:04.094+0000 7f2b838f9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T19:35:04.094 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:04.094+0000 7f2b838f9640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:35:04.094 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:04.094+0000 7f2b838f9640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:35:04.094 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:04.094+0000 7f2b838f9640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:35:04.095 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:04.095+0000 7f2b838f9640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T19:35:06.935 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T19:35:06.965 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T19:35:06.965 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T19:35:06.965 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T19:35:06.997 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T19:35:06.997 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T19:35:07.067 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T19:35:07.067 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T19:35:07.136 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T19:35:07.137 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T19:35:07.206 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T19:35:07.206 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T19:35:07.275 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T19:35:07.275 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T19:35:07.307 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T19:35:07.307 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T19:35:07.377 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T19:35:07.377 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T19:35:07.446 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T19:35:07.446 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T19:35:07.515 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T19:35:07.515 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T19:35:07.582 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T19:35:07.582 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T19:35:07.600 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T19:35:07.600 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T19:35:07.638 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T19:35:07.638 DEBUG:teuthology.orchestra.run.smithi044:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:35:07.643 DEBUG:teuthology.orchestra.run.smithi111:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:35:07.682 INFO:teuthology.orchestra.run.smithi044.stdout:[mgr.y] 2024-03-25T19:35:07.682 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDM0QFmP2N/MxAA39Mz+Om9InoCIcLVBOp1Kg== 2024-03-25T19:35:07.682 INFO:teuthology.orchestra.run.smithi044.stdout:[osd.0] 2024-03-25T19:35:07.682 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDQ0QFmmO7ZDBAALTm6iu1FxtaORPVubpjaKA== 2024-03-25T19:35:07.682 INFO:teuthology.orchestra.run.smithi044.stdout:[osd.1] 2024-03-25T19:35:07.682 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDT0QFmkEIHCxAArYkayfWxAa59SbotdDvqLQ== 2024-03-25T19:35:07.682 INFO:teuthology.orchestra.run.smithi044.stdout:[osd.2] 2024-03-25T19:35:07.682 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDW0QFmnvuZCBAAD0KLO8/2DZpquIJGd9rcbg== 2024-03-25T19:35:07.682 INFO:teuthology.orchestra.run.smithi044.stdout:[osd.3] 2024-03-25T19:35:07.682 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDZ0QFmfAQCBxAAntRxDYpAFA/lcKzVMI0u2A== 2024-03-25T19:35:07.682 INFO:teuthology.orchestra.run.smithi044.stdout:[mgr.x] 2024-03-25T19:35:07.683 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDM0QFmUZN2NxAAx8WYhMstGU6hfFrw+WoYjw== 2024-03-25T19:35:07.683 INFO:teuthology.orchestra.run.smithi044.stdout:[osd.4] 2024-03-25T19:35:07.683 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDf0QFm4FNTDBAAyP/Snxr92bCvzckT9suUBQ== 2024-03-25T19:35:07.683 INFO:teuthology.orchestra.run.smithi044.stdout:[osd.5] 2024-03-25T19:35:07.683 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDi0QFmKr+xCRAAw8s0fWQmg6pH3kjDKFC/NA== 2024-03-25T19:35:07.683 INFO:teuthology.orchestra.run.smithi044.stdout:[osd.6] 2024-03-25T19:35:07.683 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDl0QFmmGRzBxAAUblJIMmP8+1qkhnWpLOnOg== 2024-03-25T19:35:07.683 INFO:teuthology.orchestra.run.smithi044.stdout:[osd.7] 2024-03-25T19:35:07.683 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDo0QFmjmOxBRAAp6aQMJGh8wWf95AX3mgwQQ== 2024-03-25T19:35:07.683 INFO:teuthology.orchestra.run.smithi044.stdout:[client.0] 2024-03-25T19:35:07.683 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDM0QFm/JYfOhAAABz6lO669ChcUmqVCcTdPA== 2024-03-25T19:35:07.683 INFO:teuthology.orchestra.run.smithi044.stdout:[client.1] 2024-03-25T19:35:07.683 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDN0QFm2xTvARAAOsKuQIuM25Q1scelfFgm5Q== 2024-03-25T19:35:07.706 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr.y] 2024-03-25T19:35:07.706 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDM0QFmP2N/MxAA39Mz+Om9InoCIcLVBOp1Kg== 2024-03-25T19:35:07.706 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.0] 2024-03-25T19:35:07.706 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDQ0QFmmO7ZDBAALTm6iu1FxtaORPVubpjaKA== 2024-03-25T19:35:07.706 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.1] 2024-03-25T19:35:07.706 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDT0QFmkEIHCxAArYkayfWxAa59SbotdDvqLQ== 2024-03-25T19:35:07.706 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.2] 2024-03-25T19:35:07.706 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDW0QFmnvuZCBAAD0KLO8/2DZpquIJGd9rcbg== 2024-03-25T19:35:07.706 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.3] 2024-03-25T19:35:07.706 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDZ0QFmfAQCBxAAntRxDYpAFA/lcKzVMI0u2A== 2024-03-25T19:35:07.706 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr.x] 2024-03-25T19:35:07.707 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDM0QFmUZN2NxAAx8WYhMstGU6hfFrw+WoYjw== 2024-03-25T19:35:07.707 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.4] 2024-03-25T19:35:07.707 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDf0QFm4FNTDBAAyP/Snxr92bCvzckT9suUBQ== 2024-03-25T19:35:07.707 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.5] 2024-03-25T19:35:07.707 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDi0QFmKr+xCRAAw8s0fWQmg6pH3kjDKFC/NA== 2024-03-25T19:35:07.707 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.6] 2024-03-25T19:35:07.707 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDl0QFmmGRzBxAAUblJIMmP8+1qkhnWpLOnOg== 2024-03-25T19:35:07.707 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.7] 2024-03-25T19:35:07.707 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDo0QFmjmOxBRAAp6aQMJGh8wWf95AX3mgwQQ== 2024-03-25T19:35:07.707 INFO:teuthology.orchestra.run.smithi111.stdout:[client.0] 2024-03-25T19:35:07.707 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDM0QFm/JYfOhAAABz6lO669ChcUmqVCcTdPA== 2024-03-25T19:35:07.707 INFO:teuthology.orchestra.run.smithi111.stdout:[client.1] 2024-03-25T19:35:07.707 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDN0QFm2xTvARAAOsKuQIuM25Q1scelfFgm5Q== 2024-03-25T19:35:07.708 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:35:07.727 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:35:07.801 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:07.803 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:07.856 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:07.858 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:07.909 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:07.911 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:07.963 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:07.965 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:08.016 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:35:08.018 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:35:08.071 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:08.073 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:08.124 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:08.126 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:08.177 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:08.179 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:08.230 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:08.233 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:35:08.284 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:35:08.286 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:35:08.340 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:35:08.342 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:35:08.397 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T19:35:08.397 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T19:35:08.423 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:35:08.520 DEBUG:teuthology.orchestra.run.smithi044:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T19:35:08.546 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T19:35:08.612 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:35:08.707 DEBUG:teuthology.orchestra.run.smithi044:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T19:35:08.734 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T19:35:08.761 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:35:08.894 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T19:35:08.921 DEBUG:teuthology.orchestra.run.smithi044:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:35:08.923 DEBUG:teuthology.orchestra.run.smithi111:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:35:08.976 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T19:35:08.976 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:35:08.976 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T19:35:08.976 DEBUG:teuthology.orchestra.run.smithi044:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T19:35:08.979 INFO:tasks.ceph.mon.a:Started 2024-03-25T19:35:08.979 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:35:08.979 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T19:35:08.979 DEBUG:teuthology.orchestra.run.smithi044:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T19:35:08.981 INFO:tasks.ceph.mon.c:Started 2024-03-25T19:35:08.981 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:35:08.981 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T19:35:08.981 DEBUG:teuthology.orchestra.run.smithi111:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T19:35:09.019 INFO:tasks.ceph.mon.b:Started 2024-03-25T19:35:09.019 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T19:35:09.019 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T19:35:09.019 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T19:35:09.021 INFO:tasks.ceph.mgr.y:Started 2024-03-25T19:35:09.022 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T19:35:09.022 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T19:35:09.023 INFO:tasks.ceph.mgr.x:Started 2024-03-25T19:35:09.024 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T19:35:09.024 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T19:35:09.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:09.126+0000 7fb0a1fd6640 10 monclient: get_monmap_and_config 2024-03-25T19:35:09.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:09.126+0000 7fb0a1fd6640 10 monclient: build_initial_monmap 2024-03-25T19:35:09.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:09.126+0000 7fb0a1fd6640 10 monclient: monmap: 2024-03-25T19:35:09.127 INFO:teuthology.orchestra.run.smithi044.stderr:epoch 0 2024-03-25T19:35:09.127 INFO:teuthology.orchestra.run.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:09.127 INFO:teuthology.orchestra.run.smithi044.stderr:last_changed 2024-03-25T19:35:09.127923+0000 2024-03-25T19:35:09.127 INFO:teuthology.orchestra.run.smithi044.stderr:created 2024-03-25T19:35:09.127923+0000 2024-03-25T19:35:09.127 INFO:teuthology.orchestra.run.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:09.127 INFO:teuthology.orchestra.run.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:09.127 INFO:teuthology.orchestra.run.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:09.127 INFO:teuthology.orchestra.run.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:35:09.127 INFO:teuthology.orchestra.run.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:35:09.127 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T19:35:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:09.127+0000 7fb0a1fd6640 1 Processor -- start 2024-03-25T19:35:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:09.127+0000 7fb0a1fd6640 1 -- start start 2024-03-25T19:35:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:09.127+0000 7fb0a1fd6640 10 monclient: init 2024-03-25T19:35:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:09.128+0000 7fb0a1fd6640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:09.128+0000 7fb0a1fd6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:35:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:09.128+0000 7fb0a1fd6640 10 monclient(hunting): picked mon.noname-c con 0x7fb09c0b4700 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:09.128+0000 7fb0a1fd6640 10 monclient(hunting): picked mon.noname-b con 0x7fb09c142a20 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:09.128+0000 7fb0a1fd6640 10 monclient(hunting): picked mon.noname-a con 0x7fb09c0a96b0 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:09.128+0000 7fb0a1fd6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09c0a9d60 con 0x7fb09c0a96b0 2024-03-25T19:35:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:09.128+0000 7fb0a1fd6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09c13ff10 con 0x7fb09c0b4700 2024-03-25T19:35:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:09.128+0000 7fb0a1fd6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09c0aa1f0 con 0x7fb09c142a20 2024-03-25T19:35:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:09.128+0000 7fb0a1fd6640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:09.128+0000 7fb0a1fd6640 10 monclient(hunting): authenticate will time out at 5373.299033s 2024-03-25T19:35:09.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:09.128+0000 7fb09bfff640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7fb09c0a96b0 legacy=0x7fb09c13f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6789/0 2024-03-25T19:35:09.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:09.128+0000 7fb09affd640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7fb09c142a20 legacy=0x7fb09c13f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6790/0 2024-03-25T19:35:09.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:09.128+0000 7fb09b7fe640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7fb09c0b4700 legacy=0x7fb09c142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.111:6789/0 2024-03-25T19:35:09.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:09.329+0000 7fb09bfff640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7fb09c0a96b0 legacy=0x7fb09c13f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6789/0 2024-03-25T19:35:09.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:09.329+0000 7fb09affd640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7fb09c142a20 legacy=0x7fb09c13f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6790/0 2024-03-25T19:35:09.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:09.329+0000 7fb09b7fe640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7fb09c0b4700 legacy=0x7fb09c142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.111:6789/0 2024-03-25T19:35:09.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:09.730+0000 7fb09bfff640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7fb09c0a96b0 legacy=0x7fb09c13f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6789/0 2024-03-25T19:35:09.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:09.730+0000 7fb09affd640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7fb09c142a20 legacy=0x7fb09c13f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6790/0 2024-03-25T19:35:09.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:09.730+0000 7fb09b7fe640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7fb09c0b4700 legacy=0x7fb09c142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.111:6789/0 2024-03-25T19:35:10.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:10.531+0000 7fb09bfff640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7fb09c0a96b0 legacy=0x7fb09c13f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6789/0 2024-03-25T19:35:10.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:10.531+0000 7fb09affd640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7fb09c142a20 legacy=0x7fb09c13f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6790/0 2024-03-25T19:35:10.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:10.531+0000 7fb09b7fe640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7fb09c0b4700 legacy=0x7fb09c142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.111:6789/0 2024-03-25T19:35:12.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:12.128+0000 7fb0997fa640 10 monclient(hunting): tick 2024-03-25T19:35:12.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:12.128+0000 7fb0997fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:35:12.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:12.128+0000 7fb0997fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:35:12.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:12.128+0000 7fb0997fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:35:12.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:12.128+0000 7fb0997fa640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7fb09c142a20 legacy=0x7fb09c13f0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:35:12.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:12.128+0000 7fb0997fa640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7fb09c0b4700 legacy=0x7fb09c142310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:35:12.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:12.128+0000 7fb0997fa640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7fb09c0a96b0 legacy=0x7fb09c13f800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:35:12.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:12.128+0000 7fb0997fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:35:12.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:12.128+0000 7fb0997fa640 10 monclient(hunting): picked mon.noname-a con 0x7fb09c0a96b0 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:12.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:12.128+0000 7fb0997fa640 10 monclient(hunting): picked mon.noname-b con 0x7fb09c142a20 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:12.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:12.128+0000 7fb0997fa640 10 monclient(hunting): picked mon.noname-c con 0x7fb09c0b4700 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:12.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:12.128+0000 7fb0997fa640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09c0b4ae0 con 0x7fb09c0a96b0 2024-03-25T19:35:12.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:12.128+0000 7fb0997fa640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09c0a9d60 con 0x7fb09c0b4700 2024-03-25T19:35:12.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:12.128+0000 7fb09b7fe640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7fb09c0a96b0 legacy=0x7fb0880056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6789/0 2024-03-25T19:35:12.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:12.128+0000 7fb0997fa640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09c13ff10 con 0x7fb09c142a20 2024-03-25T19:35:12.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:12.128+0000 7fb0997fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:12.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:12.128+0000 7fb09affd640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7fb09c142a20 legacy=0x7fb088008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6790/0 2024-03-25T19:35:12.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:12.128+0000 7fb09bfff640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7fb09c0b4700 legacy=0x7fb08800c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.111:6789/0 2024-03-25T19:35:12.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:12.329+0000 7fb09b7fe640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7fb09c0a96b0 legacy=0x7fb0880056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6789/0 2024-03-25T19:35:12.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:12.329+0000 7fb09affd640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7fb09c142a20 legacy=0x7fb088008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6790/0 2024-03-25T19:35:12.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:12.329+0000 7fb09bfff640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7fb09c0b4700 legacy=0x7fb08800c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.111:6789/0 2024-03-25T19:35:12.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:12.730+0000 7fb09b7fe640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7fb09c0a96b0 legacy=0x7fb0880056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6789/0 2024-03-25T19:35:12.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:12.730+0000 7fb09affd640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7fb09c142a20 legacy=0x7fb088008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6790/0 2024-03-25T19:35:12.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:12.730+0000 7fb09bfff640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7fb09c0b4700 legacy=0x7fb08800c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.111:6789/0 2024-03-25T19:35:13.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:13.531+0000 7fb09b7fe640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7fb09c0a96b0 legacy=0x7fb0880056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6789/0 2024-03-25T19:35:13.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:13.531+0000 7fb09affd640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7fb09c142a20 legacy=0x7fb088008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6790/0 2024-03-25T19:35:13.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:13.531+0000 7fb09bfff640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7fb09c0b4700 legacy=0x7fb08800c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.111:6789/0 2024-03-25T19:35:15.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:15.128+0000 7fb0997fa640 10 monclient(hunting): tick 2024-03-25T19:35:15.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:15.128+0000 7fb0997fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:35:15.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:15.128+0000 7fb0997fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:35:15.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:15.128+0000 7fb0997fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:35:15.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:15.128+0000 7fb0997fa640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7fb09c142a20 legacy=0x7fb088008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:35:15.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:15.128+0000 7fb0997fa640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7fb09c0b4700 legacy=0x7fb08800c790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:35:15.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:15.128+0000 7fb0997fa640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7fb09c0a96b0 legacy=0x7fb0880056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:35:15.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:15.128+0000 7fb0997fa640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:35:15.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:15.128+0000 7fb0997fa640 10 monclient(hunting): picked mon.noname-c con 0x7fb09c0b4700 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:15.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:15.128+0000 7fb0997fa640 10 monclient(hunting): picked mon.noname-b con 0x7fb09c142a20 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:15.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:15.128+0000 7fb0997fa640 10 monclient(hunting): picked mon.noname-a con 0x7fb09c0a96b0 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:15.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:15.128+0000 7fb0997fa640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09c0aa1f0 con 0x7fb09c0a96b0 2024-03-25T19:35:15.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:15.128+0000 7fb0997fa640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09c0b4ae0 con 0x7fb09c0b4700 2024-03-25T19:35:15.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:15.128+0000 7fb0997fa640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09c0a9d60 con 0x7fb09c142a20 2024-03-25T19:35:15.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:15.128+0000 7fb0997fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:15.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:15.128+0000 7fb09bfff640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7fb09c0a96b0 legacy=0x7fb0880106b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6789/0 2024-03-25T19:35:15.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:15.128+0000 7fb09affd640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7fb09c142a20 legacy=0x7fb088008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6790/0 2024-03-25T19:35:15.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:15.128+0000 7fb09b7fe640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7fb09c0b4700 legacy=0x7fb08800ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.111:6789/0 2024-03-25T19:35:15.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:15.329+0000 7fb09bfff640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7fb09c0a96b0 legacy=0x7fb0880106b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6789/0 2024-03-25T19:35:15.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:15.329+0000 7fb09affd640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7fb09c142a20 legacy=0x7fb088008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6790/0 2024-03-25T19:35:15.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:15.329+0000 7fb09b7fe640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7fb09c0b4700 legacy=0x7fb08800ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.111:6789/0 2024-03-25T19:35:15.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:15.730+0000 7fb09affd640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7fb09c142a20 legacy=0x7fb088008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6790/0 2024-03-25T19:35:15.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:15.730+0000 7fb09bfff640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7fb09c0a96b0 legacy=0x7fb0880106b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6789/0 2024-03-25T19:35:15.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:15.730+0000 7fb09b7fe640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7fb09c0b4700 legacy=0x7fb08800ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.111:6789/0 2024-03-25T19:35:16.008 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:00:06.888 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:16.315 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:00:07.195 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:16.492 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:00:07.393 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:16.499 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:00:07.401 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:16.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:16.531+0000 7fb09affd640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7fb09c142a20 legacy=0x7fb088008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6790/0 2024-03-25T19:35:16.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:16.531+0000 7fb09bfff640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7fb09c0a96b0 legacy=0x7fb0880106b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.44:6789/0 2024-03-25T19:35:16.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:16.531+0000 7fb09b7fe640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7fb09c0b4700 legacy=0x7fb08800ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.111:6789/0 2024-03-25T19:35:16.819 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:00:07.721 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:16.827 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:00:07.729 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:18.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:18.128+0000 7fb0997fa640 10 monclient(hunting): tick 2024-03-25T19:35:18.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:18.128+0000 7fb0997fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:35:18.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:18.128+0000 7fb0997fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:35:18.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:18.128+0000 7fb0997fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:35:18.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:18.128+0000 7fb0997fa640 1 -- >> v1:172.21.15.44:6790/0 conn(0x7fb09c142a20 legacy=0x7fb088008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:35:18.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:18.128+0000 7fb0997fa640 1 -- >> v1:172.21.15.111:6789/0 conn(0x7fb09c0b4700 legacy=0x7fb08800ffa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:35:18.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:18.128+0000 7fb0997fa640 1 -- >> v1:172.21.15.44:6789/0 conn(0x7fb09c0a96b0 legacy=0x7fb0880106b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:35:18.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:18.128+0000 7fb0997fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:18.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:18.128+0000 7fb0997fa640 10 monclient(hunting): picked mon.noname-c con 0x7fb09c0b4700 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:18.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:18.128+0000 7fb0997fa640 10 monclient(hunting): picked mon.noname-a con 0x7fb09c0a96b0 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:18.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:18.128+0000 7fb0997fa640 10 monclient(hunting): picked mon.noname-b con 0x7fb09c142a20 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:18.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:18.128+0000 7fb0997fa640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09c13ff10 con 0x7fb09c0a96b0 2024-03-25T19:35:18.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:18.128+0000 7fb0997fa640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09c0aa1f0 con 0x7fb09c0b4700 2024-03-25T19:35:18.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:18.128+0000 7fb0997fa640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09c0b4ae0 con 0x7fb09c142a20 2024-03-25T19:35:18.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:18.128+0000 7fb0997fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:18.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:18.162+0000 7fb09b7fe640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb09c0b4700 0x7fb08800ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:49626/0 (socket says 172.21.15.44:49626) 2024-03-25T19:35:18.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:18.162+0000 7fb09b7fe640 1 -- 172.21.15.44:0/2166189675 learned_addr learned my addr 172.21.15.44:0/2166189675 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:35:18.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:18.523+0000 7fb09bfff640 1 -- 172.21.15.44:0/2166189675 >> v1:172.21.15.44:6790/0 conn(0x7fb09c142a20 legacy=0x7fb088008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-25T19:35:18.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:18.523+0000 7fb09bfff640 1 -- 172.21.15.44:0/2166189675 >> v1:172.21.15.44:6790/0 conn(0x7fb09c142a20 legacy=0x7fb088008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T19:35:18.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:18.523+0000 7fb09bfff640 1 --1- 172.21.15.44:0/2166189675 >> v1:172.21.15.44:6790/0 conn(0x7fb09c142a20 0x7fb088008ea0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T19:35:18.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:18.561+0000 7fb09affd640 1 -- 172.21.15.44:0/2166189675 >> v1:172.21.15.44:6789/0 conn(0x7fb09c0a96b0 legacy=0x7fb0880106b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-25T19:35:18.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:18.561+0000 7fb09affd640 1 -- 172.21.15.44:0/2166189675 >> v1:172.21.15.44:6789/0 conn(0x7fb09c0a96b0 legacy=0x7fb0880106b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T19:35:18.561 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:18.561+0000 7fb09affd640 1 --1- 172.21.15.44:0/2166189675 >> v1:172.21.15.44:6789/0 conn(0x7fb09c0a96b0 0x7fb0880106b0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T19:35:21.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:21.129+0000 7fb0997fa640 10 monclient(hunting): tick 2024-03-25T19:35:21.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:21.129+0000 7fb0997fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:35:21.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:21.129+0000 7fb0997fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:35:21.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:21.129+0000 7fb0997fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:35:21.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:21.129+0000 7fb0997fa640 1 -- 172.21.15.44:0/2166189675 >> v1:172.21.15.44:6790/0 conn(0x7fb09c142a20 legacy=0x7fb088008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:21.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:21.129+0000 7fb0997fa640 1 -- 172.21.15.44:0/2166189675 >> v1:172.21.15.111:6789/0 conn(0x7fb09c0b4700 legacy=0x7fb08800ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:21.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:21.129+0000 7fb0997fa640 1 -- 172.21.15.44:0/2166189675 >> v1:172.21.15.44:6789/0 conn(0x7fb09c0a96b0 legacy=0x7fb0880106b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:21.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:21.129+0000 7fb0997fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:21.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:21.129+0000 7fb0997fa640 10 monclient(hunting): picked mon.noname-c con 0x7fb0880056f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:21.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:21.129+0000 7fb0997fa640 10 monclient(hunting): picked mon.noname-a con 0x7fb08800cec0 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:21.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:21.129+0000 7fb0997fa640 10 monclient(hunting): picked mon.noname-b con 0x7fb09c142a20 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:21.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:21.129+0000 7fb0997fa640 1 -- 172.21.15.44:0/2166189675 --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09c0a9d60 con 0x7fb08800cec0 2024-03-25T19:35:21.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:21.129+0000 7fb0997fa640 1 -- 172.21.15.44:0/2166189675 --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb08800d2a0 con 0x7fb0880056f0 2024-03-25T19:35:21.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:21.129+0000 7fb0997fa640 1 -- 172.21.15.44:0/2166189675 --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb08800d430 con 0x7fb09c142a20 2024-03-25T19:35:21.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:21.129+0000 7fb0997fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:24.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:24.129+0000 7fb0997fa640 10 monclient(hunting): tick 2024-03-25T19:35:24.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:24.129+0000 7fb0997fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:35:24.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:24.129+0000 7fb0997fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:35:24.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:24.129+0000 7fb0997fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:35:24.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:24.129+0000 7fb0997fa640 1 -- 172.21.15.44:0/2166189675 >> v1:172.21.15.44:6790/0 conn(0x7fb09c142a20 legacy=0x7fb0880106b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:24.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:24.129+0000 7fb0997fa640 1 -- 172.21.15.44:0/2166189675 >> v1:172.21.15.111:6789/0 conn(0x7fb0880056f0 legacy=0x7fb08800c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:24.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:24.129+0000 7fb0997fa640 1 -- 172.21.15.44:0/2166189675 >> v1:172.21.15.44:6789/0 conn(0x7fb08800cec0 legacy=0x7fb08800ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:24.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:24.129+0000 7fb0997fa640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:35:24.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:24.129+0000 7fb0997fa640 10 monclient(hunting): picked mon.noname-a con 0x7fb088009040 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:24.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:24.129+0000 7fb0997fa640 10 monclient(hunting): picked mon.noname-c con 0x7fb0880056f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:24.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:24.129+0000 7fb0997fa640 10 monclient(hunting): picked mon.noname-b con 0x7fb09c142a20 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:24.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:24.129+0000 7fb0997fa640 1 -- 172.21.15.44:0/2166189675 --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb088008eb0 con 0x7fb088009040 2024-03-25T19:35:24.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:24.129+0000 7fb0997fa640 1 -- 172.21.15.44:0/2166189675 --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb08800ffa0 con 0x7fb0880056f0 2024-03-25T19:35:24.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:24.129+0000 7fb0997fa640 1 -- 172.21.15.44:0/2166189675 --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb088010130 con 0x7fb09c142a20 2024-03-25T19:35:24.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:24.129+0000 7fb0997fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:27.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.129+0000 7fb0997fa640 10 monclient(hunting): tick 2024-03-25T19:35:27.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.129+0000 7fb0997fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:35:27.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.129+0000 7fb0997fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:35:27.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.129+0000 7fb0997fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:35:27.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.129+0000 7fb0997fa640 1 -- 172.21.15.44:0/2166189675 >> v1:172.21.15.44:6790/0 conn(0x7fb09c142a20 legacy=0x7fb0880106b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:27.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.129+0000 7fb0997fa640 1 -- 172.21.15.44:0/2166189675 >> v1:172.21.15.111:6789/0 conn(0x7fb0880056f0 legacy=0x7fb08800c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:27.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.129+0000 7fb0997fa640 1 -- 172.21.15.44:0/2166189675 >> v1:172.21.15.44:6789/0 conn(0x7fb088009040 legacy=0x7fb08800d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:27.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.129+0000 7fb0997fa640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:27.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.129+0000 7fb0997fa640 10 monclient(hunting): picked mon.noname-b con 0x7fb08800df60 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:27.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.129+0000 7fb0997fa640 10 monclient(hunting): picked mon.noname-c con 0x7fb0880056f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:27.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.129+0000 7fb0997fa640 10 monclient(hunting): picked mon.noname-a con 0x7fb088009040 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:27.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.129+0000 7fb0997fa640 1 -- 172.21.15.44:0/2166189675 --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0880102c0 con 0x7fb088009040 2024-03-25T19:35:27.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.129+0000 7fb0997fa640 1 -- 172.21.15.44:0/2166189675 --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb08800fb40 con 0x7fb0880056f0 2024-03-25T19:35:27.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.129+0000 7fb0997fa640 1 -- 172.21.15.44:0/2166189675 --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb08800fd70 con 0x7fb08800df60 2024-03-25T19:35:27.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.129+0000 7fb0997fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:27.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.132+0000 7fb09affd640 1 -- 172.21.15.44:0/2166189675 >> v1:172.21.15.111:6789/0 conn(0x7fb0880056f0 legacy=0x7fb08800c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-25T19:35:27.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.132+0000 7fb09affd640 1 -- 172.21.15.44:0/2166189675 >> v1:172.21.15.111:6789/0 conn(0x7fb0880056f0 legacy=0x7fb08800c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:35:27.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.132+0000 7fb09affd640 1 --1- 172.21.15.44:0/2166189675 >> v1:172.21.15.111:6789/0 conn(0x7fb0880056f0 0x7fb08800c710 :-1 s=OPENED pgs=22 cs=1 l=1).handle_message read tag failed 2024-03-25T19:35:27.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.132+0000 7fb09affd640 1 --1- 172.21.15.44:0/2166189675 >> v1:172.21.15.111:6789/0 conn(0x7fb0880056f0 0x7fb08800c710 :-1 s=OPENED pgs=22 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:35:27.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.132+0000 7fb09a7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.111:6789/0 2024-03-25T19:35:27.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.353+0000 7fb09a7fc640 1 -- 172.21.15.44:0/2166189675 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712163630 0 0) 0x7fb0880102c0 con 0x7fb088009040 2024-03-25T19:35:27.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.353+0000 7fb09a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:27.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.353+0000 7fb09a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:27.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.353+0000 7fb09a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:27.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.353+0000 7fb09a7fc640 10 monclient(hunting): my global_id is 4101 2024-03-25T19:35:27.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.353+0000 7fb09a7fc640 1 -- 172.21.15.44:0/2166189675 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb084003520 con 0x7fb088009040 2024-03-25T19:35:27.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.368+0000 7fb09a7fc640 1 -- 172.21.15.44:0/2166189675 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3757669336 0 0) 0x7fb08800fd70 con 0x7fb08800df60 2024-03-25T19:35:27.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.368+0000 7fb09a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:27.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.368+0000 7fb09a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:27.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.368+0000 7fb09a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:27.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.368+0000 7fb09a7fc640 10 monclient(hunting): my global_id is 4100 2024-03-25T19:35:27.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.368+0000 7fb09a7fc640 1 -- 172.21.15.44:0/2166189675 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0880102c0 con 0x7fb08800df60 2024-03-25T19:35:27.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.406+0000 7fb09a7fc640 1 -- 172.21.15.44:0/2166189675 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 498240497 0 0) 0x7fb084003520 con 0x7fb088009040 2024-03-25T19:35:27.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.406+0000 7fb09a7fc640 1 -- 172.21.15.44:0/2166189675 >> v1:172.21.15.44:6790/0 conn(0x7fb08800df60 legacy=0x7fb08800e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:27.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.406+0000 7fb09a7fc640 1 -- 172.21.15.44:0/2166189675 >> v1:172.21.15.111:6789/0 conn(0x7fb0880056f0 legacy=0x7fb08800c710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:35:27.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.406+0000 7fb09a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:35:27.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.406+0000 7fb09a7fc640 1 monclient: found mon.noname-a 2024-03-25T19:35:27.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.406+0000 7fb09a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:27.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.406+0000 7fb09a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:35:27.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.406+0000 7fb09a7fc640 1 -- 172.21.15.44:0/2166189675 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0880105d0 con 0x7fb088009040 2024-03-25T19:35:27.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.406+0000 7fb09a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:35:27.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.406+0000 7fb09a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:35:27.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.406+0000 7fb09a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:27.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.406+0000 7fb09a7fc640 1 -- 172.21.15.44:0/2166189675 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fb094004480 con 0x7fb088009040 2024-03-25T19:35:27.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.406+0000 7fb0a1fd6640 5 monclient: authenticate success, global_id 4101 2024-03-25T19:35:27.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.406+0000 7fb0a1fd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:27.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.406+0000 7fb09a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:27.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.406+0000 7fb09a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:27.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.406+0000 7fb09a7fc640 10 monclient: dump: 2024-03-25T19:35:27.407 INFO:teuthology.orchestra.run.smithi044.stderr:epoch 1 2024-03-25T19:35:27.407 INFO:teuthology.orchestra.run.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:27.407 INFO:teuthology.orchestra.run.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:27.407 INFO:teuthology.orchestra.run.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:27.407 INFO:teuthology.orchestra.run.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:27.408 INFO:teuthology.orchestra.run.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:27.408 INFO:teuthology.orchestra.run.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:27.408 INFO:teuthology.orchestra.run.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:27.408 INFO:teuthology.orchestra.run.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:27.408 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T19:35:27.408 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.406+0000 7fb0a1fd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:27.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.457+0000 7fb09a7fc640 1 -- 172.21.15.44:0/2166189675 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0940045e0 con 0x7fb088009040 2024-03-25T19:35:27.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.457+0000 7fb09a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:27.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.457+0000 7fb09a7fc640 1 -- 172.21.15.44:0/2166189675 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fb094004a30 con 0x7fb088009040 2024-03-25T19:35:27.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.457+0000 7fb09a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:27.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.457+0000 7fb09a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:27.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.457+0000 7fb09a7fc640 10 monclient: dump: 2024-03-25T19:35:27.457 INFO:teuthology.orchestra.run.smithi044.stderr:epoch 1 2024-03-25T19:35:27.458 INFO:teuthology.orchestra.run.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:27.458 INFO:teuthology.orchestra.run.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:27.458 INFO:teuthology.orchestra.run.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:27.458 INFO:teuthology.orchestra.run.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:27.458 INFO:teuthology.orchestra.run.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:27.458 INFO:teuthology.orchestra.run.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:27.458 INFO:teuthology.orchestra.run.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:27.458 INFO:teuthology.orchestra.run.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:27.458 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T19:35:27.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.457+0000 7fb0a1fd6640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:27.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.457+0000 7fb0a1fd6640 4 set_mon_vals no callback set 2024-03-25T19:35:27.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.458+0000 7fb0a1fd6640 10 monclient: shutdown 2024-03-25T19:35:27.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.458+0000 7fb0a1fd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:27.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.458+0000 7fb0a1fd6640 1 -- 172.21.15.44:0/2166189675 >> v1:172.21.15.44:6789/0 conn(0x7fb088009040 legacy=0x7fb08800d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:27.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.458+0000 7fb0a1fd6640 1 -- 172.21.15.44:0/2166189675 shutdown_connections 2024-03-25T19:35:27.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.458+0000 7fb0a1fd6640 1 -- 172.21.15.44:0/2166189675 >> 172.21.15.44:0/2166189675 conn(0x7fb09c0668e0 msgr2=0x7fb09c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:27.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.458+0000 7fb0a1fd6640 1 -- 172.21.15.44:0/2166189675 shutdown_connections 2024-03-25T19:35:27.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.458+0000 7fb0a1fd6640 1 -- 172.21.15.44:0/2166189675 wait complete. 2024-03-25T19:35:27.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.459+0000 7fb0a1fd6640 10 monclient: build_initial_monmap 2024-03-25T19:35:27.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.459+0000 7fb0a1fd6640 10 monclient: monmap: 2024-03-25T19:35:27.465 INFO:teuthology.orchestra.run.smithi044.stderr:epoch 0 2024-03-25T19:35:27.465 INFO:teuthology.orchestra.run.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:27.465 INFO:teuthology.orchestra.run.smithi044.stderr:last_changed 0.000000 2024-03-25T19:35:27.465 INFO:teuthology.orchestra.run.smithi044.stderr:created 0.000000 2024-03-25T19:35:27.465 INFO:teuthology.orchestra.run.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:27.465 INFO:teuthology.orchestra.run.smithi044.stderr:election_strategy: 1 2024-03-25T19:35:27.465 INFO:teuthology.orchestra.run.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:27.465 INFO:teuthology.orchestra.run.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:35:27.465 INFO:teuthology.orchestra.run.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:35:27.465 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T19:35:27.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.459+0000 7fb0a1fd6640 1 librados: starting msgr at 2024-03-25T19:35:27.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.459+0000 7fb0a1fd6640 1 librados: starting objecter 2024-03-25T19:35:27.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.459+0000 7fb0a1fd6640 1 Processor -- start 2024-03-25T19:35:27.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.459+0000 7fb0a1fd6640 1 -- start start 2024-03-25T19:35:27.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.459+0000 7fb0a1fd6640 1 librados: setting wanted keys 2024-03-25T19:35:27.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.459+0000 7fb0a1fd6640 1 librados: calling monclient init 2024-03-25T19:35:27.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.459+0000 7fb0a1fd6640 10 monclient: init 2024-03-25T19:35:27.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.460+0000 7fb0a1fd6640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:27.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.460+0000 7fb0a1fd6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:35:27.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.460+0000 7fb0a1fd6640 10 monclient(hunting): picked mon.noname-a con 0x7fb0880056f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:27.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.460+0000 7fb0a1fd6640 10 monclient(hunting): picked mon.noname-b con 0x7fb088009040 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:27.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.460+0000 7fb0a1fd6640 10 monclient(hunting): picked mon.noname-c con 0x7fb08800df60 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:27.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.460+0000 7fb0a1fd6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09c1ca0e0 con 0x7fb0880056f0 2024-03-25T19:35:27.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.460+0000 7fb0a1fd6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09c0b15f0 con 0x7fb088009040 2024-03-25T19:35:27.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.460+0000 7fb0a1fd6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09c0af000 con 0x7fb08800df60 2024-03-25T19:35:27.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.460+0000 7fb0a1fd6640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:27.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.460+0000 7fb0a1fd6640 10 monclient(hunting): authenticate will time out at 5391.631570s 2024-03-25T19:35:27.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.461+0000 7fb09affd640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb088009040 0x7fb09c0ae8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:58636/0 (socket says 172.21.15.44:58636) 2024-03-25T19:35:27.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.461+0000 7fb09affd640 1 -- 172.21.15.44:0/1579377164 learned_addr learned my addr 172.21.15.44:0/1579377164 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:35:27.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.461+0000 7fb083fff640 10 client.?.objecter ms_handle_connect 0x7fb088009040 2024-03-25T19:35:27.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.461+0000 7fb083fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:27.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.461+0000 7fb083fff640 10 client.?.objecter ms_handle_connect 0x7fb08800df60 2024-03-25T19:35:27.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.461+0000 7fb083fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:27.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.462+0000 7fb09bfff640 1 -- 172.21.15.44:0/1579377164 >> v1:172.21.15.44:6790/0 conn(0x7fb08800df60 legacy=0x7fb09c1e2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T19:35:27.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.462+0000 7fb09bfff640 1 -- 172.21.15.44:0/1579377164 >> v1:172.21.15.44:6790/0 conn(0x7fb08800df60 legacy=0x7fb09c1e2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:35:27.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.462+0000 7fb09bfff640 1 --1- 172.21.15.44:0/1579377164 >> v1:172.21.15.44:6790/0 conn(0x7fb08800df60 0x7fb09c1e2160 :-1 s=OPENED pgs=23 cs=1 l=1).handle_message read tag failed 2024-03-25T19:35:27.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.462+0000 7fb09bfff640 1 --1- 172.21.15.44:0/1579377164 >> v1:172.21.15.44:6790/0 conn(0x7fb08800df60 0x7fb09c1e2160 :-1 s=OPENED pgs=23 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:35:27.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.462+0000 7fb083fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.44:6790/0 2024-03-25T19:35:27.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.512+0000 7fb083fff640 10 client.?.objecter ms_handle_connect 0x7fb0880056f0 2024-03-25T19:35:27.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.512+0000 7fb083fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:27.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.517+0000 7fb083fff640 1 -- 172.21.15.44:0/1579377164 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2825148142 0 0) 0x7fb09c0b15f0 con 0x7fb088009040 2024-03-25T19:35:27.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.517+0000 7fb083fff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:27.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.517+0000 7fb083fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:27.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.517+0000 7fb083fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:27.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.517+0000 7fb083fff640 10 monclient(hunting): my global_id is 4111 2024-03-25T19:35:27.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.517+0000 7fb083fff640 1 -- 172.21.15.44:0/1579377164 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb08800fd70 con 0x7fb088009040 2024-03-25T19:35:27.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.537+0000 7fb083fff640 1 -- 172.21.15.44:0/1579377164 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3857707033 0 0) 0x7fb09c1ca0e0 con 0x7fb0880056f0 2024-03-25T19:35:27.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.537+0000 7fb083fff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:27.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.537+0000 7fb083fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:27.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.537+0000 7fb083fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:27.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.537+0000 7fb083fff640 10 monclient(hunting): my global_id is 4113 2024-03-25T19:35:27.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.537+0000 7fb083fff640 1 -- 172.21.15.44:0/1579377164 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb09c0b15f0 con 0x7fb0880056f0 2024-03-25T19:35:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.599+0000 7fb083fff640 1 -- 172.21.15.44:0/1579377164 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3292663412 0 0) 0x7fb09c0b15f0 con 0x7fb0880056f0 2024-03-25T19:35:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.599+0000 7fb083fff640 1 -- 172.21.15.44:0/1579377164 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb09c1ca0e0 con 0x7fb0880056f0 2024-03-25T19:35:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.599+0000 7fb083fff640 1 -- 172.21.15.44:0/1579377164 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fb08c004160 con 0x7fb0880056f0 2024-03-25T19:35:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.599+0000 7fb083fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:27.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.599+0000 7fb083fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:27.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.599+0000 7fb083fff640 10 monclient(hunting): dump: 2024-03-25T19:35:27.601 INFO:teuthology.orchestra.run.smithi044.stderr:epoch 1 2024-03-25T19:35:27.601 INFO:teuthology.orchestra.run.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:27.601 INFO:teuthology.orchestra.run.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:27.601 INFO:teuthology.orchestra.run.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:27.601 INFO:teuthology.orchestra.run.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:27.601 INFO:teuthology.orchestra.run.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:27.601 INFO:teuthology.orchestra.run.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:27.602 INFO:teuthology.orchestra.run.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:27.602 INFO:teuthology.orchestra.run.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:27.602 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T19:35:27.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.600+0000 7fb083fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:27.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.614+0000 7fb083fff640 1 -- 172.21.15.44:0/1579377164 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3087218075 0 0) 0x7fb09c1ca0e0 con 0x7fb0880056f0 2024-03-25T19:35:27.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.614+0000 7fb083fff640 1 -- 172.21.15.44:0/1579377164 >> v1:172.21.15.44:6790/0 conn(0x7fb08800df60 legacy=0x7fb09c1e2160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:35:27.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.614+0000 7fb083fff640 1 -- 172.21.15.44:0/1579377164 >> v1:172.21.15.111:6789/0 conn(0x7fb088009040 legacy=0x7fb09c0ae8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:27.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.614+0000 7fb083fff640 10 monclient: _finish_hunting 0 2024-03-25T19:35:27.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.614+0000 7fb083fff640 1 monclient: found mon.a 2024-03-25T19:35:27.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.614+0000 7fb083fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:27.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.614+0000 7fb083fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:27.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.614+0000 7fb083fff640 1 -- 172.21.15.44:0/1579377164 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09c0af190 con 0x7fb0880056f0 2024-03-25T19:35:27.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.614+0000 7fb083fff640 10 monclient: _finish_auth 0 2024-03-25T19:35:27.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.614+0000 7fb083fff640 10 monclient: _check_auth_tickets 2024-03-25T19:35:27.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.614+0000 7fb083fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:27.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.614+0000 7fb0a1fd6640 5 monclient: authenticate success, global_id 4113 2024-03-25T19:35:27.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.614+0000 7fb0a1fd6640 10 monclient: _renew_subs 2024-03-25T19:35:27.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.614+0000 7fb0a1fd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:27.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.614+0000 7fb0a1fd6640 1 -- 172.21.15.44:0/1579377164 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb09c1cef30 con 0x7fb0880056f0 2024-03-25T19:35:27.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.614+0000 7fb0a1fd6640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:27.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.614+0000 7fb0a1fd6640 10 monclient: _renew_subs 2024-03-25T19:35:27.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.614+0000 7fb0a1fd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:27.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.614+0000 7fb0a1fd6640 1 -- 172.21.15.44:0/1579377164 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb09c1e3880 con 0x7fb0880056f0 2024-03-25T19:35:27.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.614+0000 7fb0a1fd6640 1 librados: init done 2024-03-25T19:35:27.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.615+0000 7fb0a1fd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:27.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.615+0000 7fb0a1fd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:27.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.615+0000 7fb0a1fd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:27.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.615+0000 7fb0a1fd6640 1 -- 172.21.15.44:0/1579377164 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb09c1c8da0 con 0x7fb0880056f0 2024-03-25T19:35:27.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.790+0000 7fb083fff640 1 -- 172.21.15.44:0/1579377164 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb08c004570 con 0x7fb0880056f0 2024-03-25T19:35:27.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.790+0000 7fb083fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:27.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.790+0000 7fb083fff640 1 -- 172.21.15.44:0/1579377164 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fb08c004940 con 0x7fb0880056f0 2024-03-25T19:35:27.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.790+0000 7fb083fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:27.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.790+0000 7fb083fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:27.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.790+0000 7fb083fff640 10 monclient: dump: 2024-03-25T19:35:27.791 INFO:teuthology.orchestra.run.smithi044.stderr:epoch 1 2024-03-25T19:35:27.791 INFO:teuthology.orchestra.run.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:27.791 INFO:teuthology.orchestra.run.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:27.791 INFO:teuthology.orchestra.run.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:27.791 INFO:teuthology.orchestra.run.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:27.792 INFO:teuthology.orchestra.run.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:27.792 INFO:teuthology.orchestra.run.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:27.792 INFO:teuthology.orchestra.run.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:27.792 INFO:teuthology.orchestra.run.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:27.792 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T19:35:27.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.790+0000 7fb083fff640 1 -- 172.21.15.44:0/1579377164 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb08c004ef0 con 0x7fb0880056f0 2024-03-25T19:35:27.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.790+0000 7fb083fff640 1 -- 172.21.15.44:0/1579377164 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 101281343 0 0) 0x7fb08c0063e0 con 0x7fb0880056f0 2024-03-25T19:35:27.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.790+0000 7fb083fff640 10 client.4113.objecter ms_dispatch 0x7fb09c000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T19:35:27.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.790+0000 7fb083fff640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T19:35:27.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.790+0000 7fb083fff640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-25T19:35:27.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.790+0000 7fb099ffb640 4 set_mon_vals no callback set 2024-03-25T19:35:27.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.790+0000 7fb083fff640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-25T19:35:27.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.790+0000 7fb083fff640 1 -- 172.21.15.44:0/1579377164 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb08c006820 con 0x7fb0880056f0 2024-03-25T19:35:27.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.790+0000 7fb083fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:27.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.790+0000 7fb083fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:27.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.858+0000 7fb0a1fd6640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:35:27.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.858+0000 7fb0a1fd6640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:35:27.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.858+0000 7fb0a1fd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:27.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:27.858+0000 7fb0a1fd6640 1 -- 172.21.15.44:0/1579377164 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fb09c1c8da0 con 0x7fb0880056f0 2024-03-25T19:35:28.034 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:28.035+0000 7f882064b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:35:28.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:28.046+0000 7fb083fff640 1 -- 172.21.15.44:0/1579377164 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fb08c005280 con 0x7fb0880056f0 2024-03-25T19:35:28.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:28.046+0000 7fb083fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:35:28.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:28.046+0000 7fb083fff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T19:35:28.047 INFO:teuthology.orchestra.run.smithi044.stderr:adjusted tunables profile to default 2024-03-25T19:35:28.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:28.047+0000 7fb0a1fd6640 10 librados: watch_flush enter 2024-03-25T19:35:28.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:28.047+0000 7fb0a1fd6640 10 librados: watch_flush exit 2024-03-25T19:35:28.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:28.047+0000 7fb0a1fd6640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-25T19:35:28.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:28.047+0000 7fb0a1fd6640 10 client.4113.objecter successfully canceled tick 2024-03-25T19:35:28.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:28.047+0000 7fb0a1fd6640 10 monclient: shutdown 2024-03-25T19:35:28.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:28.047+0000 7fb0a1fd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:28.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:28.047+0000 7fb0a1fd6640 1 -- 172.21.15.44:0/1579377164 >> v1:172.21.15.44:6789/0 conn(0x7fb0880056f0 legacy=0x7fb09c1c8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:28.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:28.047+0000 7fb0a1fd6640 1 -- 172.21.15.44:0/1579377164 shutdown_connections 2024-03-25T19:35:28.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:28.047+0000 7fb0a1fd6640 1 -- 172.21.15.44:0/1579377164 >> 172.21.15.44:0/1579377164 conn(0x7fb09c0668e0 msgr2=0x7fb09c13d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:28.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:28.047+0000 7fb0a1fd6640 1 -- 172.21.15.44:0/1579377164 shutdown_connections 2024-03-25T19:35:28.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:28.047+0000 7fb0a1fd6640 1 -- 172.21.15.44:0/1579377164 wait complete. 2024-03-25T19:35:28.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:28.047+0000 7fb0a1fd6640 1 librados: shutdown 2024-03-25T19:35:28.059 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T19:35:28.059 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T19:35:28.059 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T19:35:28.059 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T19:35:28.087 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T19:35:28.087 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T19:35:28.091 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:28.093+0000 7f882064b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:35:28.092 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T19:35:28.092+0000 7f54ee4c7040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:35:28.119 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T19:35:28.119 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T19:35:28.169 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T19:35:28.170+0000 7f54ee4c7040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:35:28.186 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-03-25T19:35:28.187 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T19:35:28.255 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T19:35:28.255 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T19:35:28.282 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T19:35:28.282 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T19:35:28.349 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T19:35:28.349 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T19:35:28.416 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-25T19:35:28.417 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T19:35:28.483 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new 6c91f3d4-f50d-4bf8-92a7-d50969121e79 0 2024-03-25T19:35:28.624 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.625+0000 7f341e4e4640 10 monclient: get_monmap_and_config 2024-03-25T19:35:28.624 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.625+0000 7f341e4e4640 10 monclient: build_initial_monmap 2024-03-25T19:35:28.624 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.625+0000 7f341e4e4640 10 monclient: monmap: 2024-03-25T19:35:28.624 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T19:35:28.624 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:28.624 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:35:28.625835+0000 2024-03-25T19:35:28.625 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:35:28.625835+0000 2024-03-25T19:35:28.625 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:28.625 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:28.625 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:28.625 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:35:28.625 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:35:28.625 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:28.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.626+0000 7f341e4e4640 1 Processor -- start 2024-03-25T19:35:28.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.626+0000 7f341e4e4640 1 -- start start 2024-03-25T19:35:28.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.626+0000 7f341e4e4640 10 monclient: init 2024-03-25T19:35:28.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.627+0000 7f341e4e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:28.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.627+0000 7f341e4e4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:35:28.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.627+0000 7f341e4e4640 10 monclient(hunting): picked mon.noname-c con 0x7f34180b4700 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:28.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.627+0000 7f341e4e4640 10 monclient(hunting): picked mon.noname-b con 0x7f3418142a20 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:28.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.627+0000 7f341e4e4640 10 monclient(hunting): picked mon.noname-a con 0x7f34180a96b0 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:28.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.627+0000 7f341e4e4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34180a9d60 con 0x7f34180a96b0 2024-03-25T19:35:28.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.627+0000 7f341e4e4640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f341813ff10 con 0x7f34180b4700 2024-03-25T19:35:28.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.627+0000 7f341e4e4640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34180aa1f0 con 0x7f3418142a20 2024-03-25T19:35:28.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.627+0000 7f341e4e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:28.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.627+0000 7f341e4e4640 10 monclient(hunting): authenticate will time out at 5349.761963s 2024-03-25T19:35:28.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.627+0000 7f34177fe640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f3418142a20 0x7f341813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.111:42362/0 (socket says 172.21.15.111:42362) 2024-03-25T19:35:28.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.627+0000 7f3417fff640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f34180b4700 0x7f3418142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:47206/0 (socket says 172.21.15.111:47206) 2024-03-25T19:35:28.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.627+0000 7f34177fe640 1 -- 172.21.15.111:0/2971595552 learned_addr learned my addr 172.21.15.111:0/2971595552 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T19:35:28.633 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.635+0000 7f3416ffd640 1 -- 172.21.15.111:0/2971595552 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800136895 0 0) 0x7f34180a9d60 con 0x7f34180a96b0 2024-03-25T19:35:28.634 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.635+0000 7f3416ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:28.634 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.635+0000 7f3416ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:28.634 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.635+0000 7f3416ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:28.634 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.635+0000 7f3416ffd640 10 monclient(hunting): my global_id is 4116 2024-03-25T19:35:28.634 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.635+0000 7f3416ffd640 1 -- 172.21.15.111:0/2971595552 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3400003520 con 0x7f34180a96b0 2024-03-25T19:35:28.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.638+0000 7f3416ffd640 1 -- 172.21.15.111:0/2971595552 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2420228087 0 0) 0x7f3400003520 con 0x7f34180a96b0 2024-03-25T19:35:28.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.638+0000 7f3416ffd640 1 -- 172.21.15.111:0/2971595552 >> v1:172.21.15.44:6790/0 conn(0x7f3418142a20 legacy=0x7f341813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:28.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.638+0000 7f3416ffd640 1 -- 172.21.15.111:0/2971595552 >> v1:172.21.15.111:6789/0 conn(0x7f34180b4700 legacy=0x7f3418142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:28.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.638+0000 7f3416ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:35:28.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.638+0000 7f3416ffd640 1 monclient: found mon.noname-a 2024-03-25T19:35:28.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.638+0000 7f3416ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:28.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.638+0000 7f3416ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:35:28.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.638+0000 7f3416ffd640 1 -- 172.21.15.111:0/2971595552 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34180b4ae0 con 0x7f34180a96b0 2024-03-25T19:35:28.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.638+0000 7f3416ffd640 10 monclient: _finish_auth 0 2024-03-25T19:35:28.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.638+0000 7f3416ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:35:28.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.638+0000 7f3416ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:28.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.638+0000 7f3416ffd640 1 -- 172.21.15.111:0/2971595552 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3410001a10 con 0x7f34180a96b0 2024-03-25T19:35:28.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.638+0000 7f341e4e4640 5 monclient: authenticate success, global_id 4116 2024-03-25T19:35:28.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.638+0000 7f341e4e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:28.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.638+0000 7f3416ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:28.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.638+0000 7f3416ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:28.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.638+0000 7f3416ffd640 10 monclient: dump: 2024-03-25T19:35:28.639 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:28.639 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:28.639 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:28.639 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:28.639 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:28.639 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:28.639 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:28.639 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:28.639 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:28.639 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:28.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.638+0000 7f341e4e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:28.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.640+0000 7f3416ffd640 1 -- 172.21.15.111:0/2971595552 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3410001b70 con 0x7f34180a96b0 2024-03-25T19:35:28.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.640+0000 7f3416ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:28.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.640+0000 7f3416ffd640 1 -- 172.21.15.111:0/2971595552 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3410001fc0 con 0x7f34180a96b0 2024-03-25T19:35:28.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.640+0000 7f3416ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:28.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.640+0000 7f3416ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:28.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.640+0000 7f3416ffd640 10 monclient: dump: 2024-03-25T19:35:28.643 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:28.643 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:28.643 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:28.643 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:28.643 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:28.644 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:28.644 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:28.644 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:28.644 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:28.644 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:28.644 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.640+0000 7f341e4e4640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:28.644 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.640+0000 7f341e4e4640 4 set_mon_vals no callback set 2024-03-25T19:35:28.644 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.641+0000 7f341e4e4640 10 monclient: shutdown 2024-03-25T19:35:28.644 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.641+0000 7f341e4e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:28.644 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.641+0000 7f341e4e4640 1 -- 172.21.15.111:0/2971595552 >> v1:172.21.15.44:6789/0 conn(0x7f34180a96b0 legacy=0x7f341813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:28.644 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.641+0000 7f341e4e4640 1 -- 172.21.15.111:0/2971595552 shutdown_connections 2024-03-25T19:35:28.644 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.641+0000 7f341e4e4640 1 -- 172.21.15.111:0/2971595552 >> 172.21.15.111:0/2971595552 conn(0x7f34180668e0 msgr2=0x7f3418066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:28.644 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.641+0000 7f341e4e4640 1 -- 172.21.15.111:0/2971595552 shutdown_connections 2024-03-25T19:35:28.644 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.641+0000 7f341e4e4640 1 -- 172.21.15.111:0/2971595552 wait complete. 2024-03-25T19:35:28.644 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.642+0000 7f341e4e4640 10 monclient: build_initial_monmap 2024-03-25T19:35:28.644 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.642+0000 7f341e4e4640 10 monclient: monmap: 2024-03-25T19:35:28.645 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T19:35:28.645 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:28.645 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 0.000000 2024-03-25T19:35:28.645 INFO:teuthology.orchestra.run.smithi111.stderr:created 0.000000 2024-03-25T19:35:28.645 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:28.645 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 1 2024-03-25T19:35:28.645 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:28.645 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:35:28.645 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:35:28.645 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:28.645 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.642+0000 7f341e4e4640 1 librados: starting msgr at 2024-03-25T19:35:28.645 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.642+0000 7f341e4e4640 1 librados: starting objecter 2024-03-25T19:35:28.645 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.642+0000 7f341e4e4640 1 Processor -- start 2024-03-25T19:35:28.646 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:28.643+0000 7f882064b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:35:28.647 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.643+0000 7f341e4e4640 1 -- start start 2024-03-25T19:35:28.647 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.643+0000 7f341e4e4640 1 librados: setting wanted keys 2024-03-25T19:35:28.648 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.643+0000 7f341e4e4640 1 librados: calling monclient init 2024-03-25T19:35:28.648 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.643+0000 7f341e4e4640 10 monclient: init 2024-03-25T19:35:28.648 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.644+0000 7f341e4e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:28.648 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.644+0000 7f341e4e4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:35:28.648 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.644+0000 7f341e4e4640 10 monclient(hunting): picked mon.noname-b con 0x7f34180a96b0 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:28.648 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.644+0000 7f341e4e4640 10 monclient(hunting): picked mon.noname-c con 0x7f34180b4700 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:28.648 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.644+0000 7f341e4e4640 10 monclient(hunting): picked mon.noname-a con 0x7f3418142a20 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:28.648 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.644+0000 7f341e4e4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34180ae0b0 con 0x7f3418142a20 2024-03-25T19:35:28.648 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.644+0000 7f341e4e4640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34180ae320 con 0x7f34180a96b0 2024-03-25T19:35:28.648 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.644+0000 7f341e4e4640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34181dab00 con 0x7f34180b4700 2024-03-25T19:35:28.648 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.644+0000 7f341e4e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:28.648 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.644+0000 7f341e4e4640 10 monclient(hunting): authenticate will time out at 5349.778909s 2024-03-25T19:35:28.648 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.644+0000 7f341ca81640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f3418142a20 0x7f34181d45e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.111:44242/0 (socket says 172.21.15.111:44242) 2024-03-25T19:35:28.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.644+0000 7f341ca81640 1 -- 172.21.15.111:0/2605005824 learned_addr learned my addr 172.21.15.111:0/2605005824 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T19:35:28.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.645+0000 7f3414ff9640 10 client.?.objecter ms_handle_connect 0x7f3418142a20 2024-03-25T19:35:28.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.645+0000 7f3414ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:28.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.647+0000 7f3414ff9640 1 -- 172.21.15.111:0/2605005824 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244968126 0 0) 0x7f34180ae0b0 con 0x7f3418142a20 2024-03-25T19:35:28.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.647+0000 7f3414ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:28.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.647+0000 7f3414ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:28.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.647+0000 7f3414ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:28.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.647+0000 7f3414ff9640 10 monclient(hunting): my global_id is 4119 2024-03-25T19:35:28.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.647+0000 7f3414ff9640 1 -- 172.21.15.111:0/2605005824 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3404004eb0 con 0x7f3418142a20 2024-03-25T19:35:28.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.652+0000 7f3414ff9640 10 client.?.objecter ms_handle_connect 0x7f34180a96b0 2024-03-25T19:35:28.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.652+0000 7f3414ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:28.651 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.652+0000 7f3414ff9640 1 -- 172.21.15.111:0/2605005824 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2297204080 0 0) 0x7f3404004eb0 con 0x7f3418142a20 2024-03-25T19:35:28.651 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.652+0000 7f3414ff9640 1 -- 172.21.15.111:0/2605005824 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34180ae0b0 con 0x7f3418142a20 2024-03-25T19:35:28.651 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.652+0000 7f3414ff9640 1 -- 172.21.15.111:0/2605005824 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3410001fc0 con 0x7f3418142a20 2024-03-25T19:35:28.652 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.652+0000 7f3414ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:28.652 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.652+0000 7f3414ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:28.652 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.652+0000 7f3414ff9640 10 monclient(hunting): dump: 2024-03-25T19:35:28.652 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:28.652 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:28.652 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:28.652 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:28.652 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:28.652 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:28.652 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:28.652 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:28.652 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:28.652 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:28.653 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.652+0000 7f3414ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:28.653 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.653+0000 7f3414ff9640 10 client.?.objecter ms_handle_connect 0x7f34180b4700 2024-03-25T19:35:28.653 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.653+0000 7f3414ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:28.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.654+0000 7f3414ff9640 1 -- 172.21.15.111:0/2605005824 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 489092951 0 0) 0x7f34180ae320 con 0x7f34180a96b0 2024-03-25T19:35:28.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.654+0000 7f3414ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:28.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.654+0000 7f3414ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:28.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.654+0000 7f3414ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:28.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.654+0000 7f3414ff9640 10 monclient(hunting): my global_id is 4117 2024-03-25T19:35:28.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.654+0000 7f3414ff9640 1 -- 172.21.15.111:0/2605005824 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3404004eb0 con 0x7f34180a96b0 2024-03-25T19:35:28.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.655+0000 7f3414ff9640 1 -- 172.21.15.111:0/2605005824 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1436813926 0 0) 0x7f34181dab00 con 0x7f34180b4700 2024-03-25T19:35:28.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.655+0000 7f3414ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:28.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.655+0000 7f3414ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:28.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.655+0000 7f3414ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:28.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.655+0000 7f3414ff9640 10 monclient(hunting): my global_id is 4118 2024-03-25T19:35:28.654 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.655+0000 7f3414ff9640 1 -- 172.21.15.111:0/2605005824 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34180ae320 con 0x7f34180b4700 2024-03-25T19:35:28.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.665+0000 7f3414ff9640 1 -- 172.21.15.111:0/2605005824 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2020957509 0 0) 0x7f34180ae0b0 con 0x7f3418142a20 2024-03-25T19:35:28.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.665+0000 7f3414ff9640 1 -- 172.21.15.111:0/2605005824 >> v1:172.21.15.44:6790/0 conn(0x7f34180b4700 legacy=0x7f34180ac6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:28.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.665+0000 7f3414ff9640 1 -- 172.21.15.111:0/2605005824 >> v1:172.21.15.111:6789/0 conn(0x7f34180a96b0 legacy=0x7f34181db020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:28.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.665+0000 7f3414ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:35:28.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.665+0000 7f3414ff9640 1 monclient: found mon.a 2024-03-25T19:35:28.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.665+0000 7f3414ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:28.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.665+0000 7f3414ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:28.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.665+0000 7f3414ff9640 1 -- 172.21.15.111:0/2605005824 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34181db730 con 0x7f3418142a20 2024-03-25T19:35:28.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.665+0000 7f3414ff9640 10 monclient: _finish_auth 0 2024-03-25T19:35:28.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.665+0000 7f3414ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:35:28.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.665+0000 7f3414ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:28.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.665+0000 7f341e4e4640 5 monclient: authenticate success, global_id 4119 2024-03-25T19:35:28.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.665+0000 7f341e4e4640 10 monclient: _renew_subs 2024-03-25T19:35:28.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.665+0000 7f341e4e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:28.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.665+0000 7f341e4e4640 1 -- 172.21.15.111:0/2605005824 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34181d3fd0 con 0x7f3418142a20 2024-03-25T19:35:28.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.666+0000 7f341e4e4640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:28.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.666+0000 7f341e4e4640 10 monclient: _renew_subs 2024-03-25T19:35:28.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.666+0000 7f341e4e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:28.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.666+0000 7f341e4e4640 1 -- 172.21.15.111:0/2605005824 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34180acf90 con 0x7f3418142a20 2024-03-25T19:35:28.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.666+0000 7f341e4e4640 1 librados: init done 2024-03-25T19:35:28.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.666+0000 7f341e4e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:28.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.666+0000 7f341e4e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:28.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.666+0000 7f341e4e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:28.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.666+0000 7f341e4e4640 1 -- 172.21.15.111:0/2605005824 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34181d4210 con 0x7f3418142a20 2024-03-25T19:35:28.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.670+0000 7f3414ff9640 1 -- 172.21.15.111:0/2605005824 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3410003c20 con 0x7f3418142a20 2024-03-25T19:35:28.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.670+0000 7f3414ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:28.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.670+0000 7f3414ff9640 1 -- 172.21.15.111:0/2605005824 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3410003e50 con 0x7f3418142a20 2024-03-25T19:35:28.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.670+0000 7f3414ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:28.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.670+0000 7f3414ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:28.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.670+0000 7f3414ff9640 10 monclient: dump: 2024-03-25T19:35:28.669 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:28.669 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:28.670 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:28.670 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:28.670 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:28.670 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:28.670 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:28.670 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:28.670 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:28.670 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:28.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.670+0000 7f3414ff9640 1 -- 172.21.15.111:0/2605005824 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3410004400 con 0x7f3418142a20 2024-03-25T19:35:28.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.670+0000 7f3414ff9640 1 -- 172.21.15.111:0/2605005824 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 1739401641 0 0) 0x7f3410006910 con 0x7f3418142a20 2024-03-25T19:35:28.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.670+0000 7f3414ff9640 10 client.4119.objecter ms_dispatch 0x7f3418000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T19:35:28.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.670+0000 7f34167fc640 4 set_mon_vals no callback set 2024-03-25T19:35:28.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.670+0000 7f3414ff9640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T19:35:28.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.670+0000 7f3414ff9640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T19:35:28.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.670+0000 7f3414ff9640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T19:35:28.698 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:28.700+0000 7f882064b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:35:28.744 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:28.746+0000 7f882064b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:35:28.759 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T19:35:28.759+0000 7f54ee4c7040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:35:28.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.770+0000 7f3414ff9640 1 -- 172.21.15.111:0/2605005824 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f34100046b0 con 0x7f3418142a20 2024-03-25T19:35:28.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.770+0000 7f3414ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:28.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.770+0000 7f3414ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:28.799 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:28.800+0000 7f882064b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:35:28.815 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T19:35:28.815+0000 7f54ee4c7040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:35:28.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.842+0000 7f341e4e4640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6c91f3d4-f50d-4bf8-92a7-d50969121e79", "id": 0}] 2024-03-25T19:35:28.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.842+0000 7f341e4e4640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6c91f3d4-f50d-4bf8-92a7-d50969121e79", "id": 0}] 2024-03-25T19:35:28.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.842+0000 7f341e4e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:28.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.842+0000 7f341e4e4640 1 -- 172.21.15.111:0/2605005824 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6c91f3d4-f50d-4bf8-92a7-d50969121e79", "id": 0} v 0) v1 -- 0x7f34181dac90 con 0x7f3418142a20 2024-03-25T19:35:28.842 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:28.844+0000 7f882064b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:35:28.862 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T19:35:28.862+0000 7f54ee4c7040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:35:28.920 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T19:35:28.920+0000 7f54ee4c7040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:35:28.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.936+0000 7f3414ff9640 1 -- 172.21.15.111:0/2605005824 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6c91f3d4-f50d-4bf8-92a7-d50969121e79", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3313470928 0 732707515) 0x7f3410005b00 con 0x7f3418142a20 2024-03-25T19:35:28.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.936+0000 7f3414ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6c91f3d4-f50d-4bf8-92a7-d50969121e79", "id": 0}] 2024-03-25T19:35:28.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.936+0000 7f3414ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:28.935 INFO:teuthology.orchestra.run.smithi111.stdout:0 2024-03-25T19:35:28.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.937+0000 7f341e4e4640 10 librados: watch_flush enter 2024-03-25T19:35:28.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.937+0000 7f341e4e4640 10 librados: watch_flush exit 2024-03-25T19:35:28.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.937+0000 7f341e4e4640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T19:35:28.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.937+0000 7f341e4e4640 10 client.4119.objecter successfully canceled tick 2024-03-25T19:35:28.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.937+0000 7f341e4e4640 10 monclient: shutdown 2024-03-25T19:35:28.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.937+0000 7f341e4e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:28.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.937+0000 7f341e4e4640 1 -- 172.21.15.111:0/2605005824 >> v1:172.21.15.44:6789/0 conn(0x7f3418142a20 legacy=0x7f34181d45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:28.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.937+0000 7f341e4e4640 1 -- 172.21.15.111:0/2605005824 shutdown_connections 2024-03-25T19:35:28.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.937+0000 7f341e4e4640 1 -- 172.21.15.111:0/2605005824 >> 172.21.15.111:0/2605005824 conn(0x7f34180668e0 msgr2=0x7f34180b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:28.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.937+0000 7f341e4e4640 1 -- 172.21.15.111:0/2605005824 shutdown_connections 2024-03-25T19:35:28.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.937+0000 7f341e4e4640 1 -- 172.21.15.111:0/2605005824 wait complete. 2024-03-25T19:35:28.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:28.937+0000 7f341e4e4640 1 librados: shutdown 2024-03-25T19:35:28.946 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new 24b49d5b-ed7c-4388-a11d-0c292ef38dbd 1 2024-03-25T19:35:28.959 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:28.960+0000 7f882064b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:35:28.968 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T19:35:28.969+0000 7f54ee4c7040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:35:29.033 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:29.035+0000 7f882064b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:35:29.046 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.047+0000 7fc90ffcf640 10 monclient: get_monmap_and_config 2024-03-25T19:35:29.046 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.047+0000 7fc90ffcf640 10 monclient: build_initial_monmap 2024-03-25T19:35:29.046 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.047+0000 7fc90ffcf640 10 monclient: monmap: 2024-03-25T19:35:29.046 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T19:35:29.046 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:29.046 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:35:29.047843+0000 2024-03-25T19:35:29.046 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:35:29.047843+0000 2024-03-25T19:35:29.047 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:29.047 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:29.047 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:29.047 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:35:29.047 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:35:29.047 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:29.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.048+0000 7fc90ffcf640 1 Processor -- start 2024-03-25T19:35:29.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.048+0000 7fc90ffcf640 1 -- start start 2024-03-25T19:35:29.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.048+0000 7fc90ffcf640 10 monclient: init 2024-03-25T19:35:29.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.049+0000 7fc90ffcf640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:29.049 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.049+0000 7fc90ffcf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:35:29.049 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.049+0000 7fc90ffcf640 10 monclient(hunting): picked mon.noname-b con 0x7fc9080b4700 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:29.049 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.049+0000 7fc90ffcf640 10 monclient(hunting): picked mon.noname-a con 0x7fc908142a20 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:29.049 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.049+0000 7fc90ffcf640 10 monclient(hunting): picked mon.noname-c con 0x7fc9080a96b0 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:29.049 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.049+0000 7fc90ffcf640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9080a9d60 con 0x7fc908142a20 2024-03-25T19:35:29.049 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.049+0000 7fc90ffcf640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc90813ff10 con 0x7fc9080a96b0 2024-03-25T19:35:29.049 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.049+0000 7fc90ffcf640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9080aa1f0 con 0x7fc9080b4700 2024-03-25T19:35:29.049 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.049+0000 7fc90ffcf640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:29.049 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.049+0000 7fc90ffcf640 10 monclient(hunting): authenticate will time out at 5350.183968s 2024-03-25T19:35:29.049 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.049+0000 7fc90dd6b640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fc9080b4700 0x7fc908142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.111:42384/0 (socket says 172.21.15.111:42384) 2024-03-25T19:35:29.049 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.049+0000 7fc90dd6b640 1 -- 172.21.15.111:0/2385940262 learned_addr learned my addr 172.21.15.111:0/2385940262 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T19:35:29.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.052+0000 7fc90cd69640 1 -- 172.21.15.111:0/2385940262 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3254132276 0 0) 0x7fc9080aa1f0 con 0x7fc9080b4700 2024-03-25T19:35:29.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.052+0000 7fc90cd69640 10 monclient(hunting): _negotiate 2024-03-25T19:35:29.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.052+0000 7fc90cd69640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:29.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.052+0000 7fc90cd69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:29.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.052+0000 7fc90cd69640 10 monclient(hunting): my global_id is 4121 2024-03-25T19:35:29.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.052+0000 7fc90cd69640 1 -- 172.21.15.111:0/2385940262 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f4003520 con 0x7fc9080b4700 2024-03-25T19:35:29.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.052+0000 7fc90cd69640 1 -- 172.21.15.111:0/2385940262 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455364425 0 0) 0x7fc90813ff10 con 0x7fc9080a96b0 2024-03-25T19:35:29.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.052+0000 7fc90cd69640 10 monclient(hunting): _negotiate 2024-03-25T19:35:29.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.052+0000 7fc90cd69640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:29.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.052+0000 7fc90cd69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:29.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.052+0000 7fc90cd69640 10 monclient(hunting): my global_id is 4120 2024-03-25T19:35:29.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.052+0000 7fc90cd69640 1 -- 172.21.15.111:0/2385940262 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9080aa1f0 con 0x7fc9080a96b0 2024-03-25T19:35:29.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.052+0000 7fc90cd69640 1 -- 172.21.15.111:0/2385940262 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4151091375 0 0) 0x7fc9080a9d60 con 0x7fc908142a20 2024-03-25T19:35:29.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.052+0000 7fc90cd69640 10 monclient(hunting): _negotiate 2024-03-25T19:35:29.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.052+0000 7fc90cd69640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:29.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.052+0000 7fc90cd69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:29.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.052+0000 7fc90cd69640 10 monclient(hunting): my global_id is 4122 2024-03-25T19:35:29.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.052+0000 7fc90cd69640 1 -- 172.21.15.111:0/2385940262 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc90813ff10 con 0x7fc908142a20 2024-03-25T19:35:29.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.055+0000 7fc90cd69640 1 -- 172.21.15.111:0/2385940262 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 906698460 0 0) 0x7fc8f4003520 con 0x7fc9080b4700 2024-03-25T19:35:29.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.055+0000 7fc90cd69640 1 -- 172.21.15.111:0/2385940262 >> v1:172.21.15.111:6789/0 conn(0x7fc9080a96b0 legacy=0x7fc90813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:29.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.055+0000 7fc90cd69640 1 -- 172.21.15.111:0/2385940262 >> v1:172.21.15.44:6789/0 conn(0x7fc908142a20 legacy=0x7fc90813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:29.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.055+0000 7fc90cd69640 10 monclient: _finish_hunting 0 2024-03-25T19:35:29.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.055+0000 7fc90cd69640 1 monclient: found mon.noname-b 2024-03-25T19:35:29.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.055+0000 7fc90cd69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:29.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.055+0000 7fc90cd69640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:35:29.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.055+0000 7fc90cd69640 1 -- 172.21.15.111:0/2385940262 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9080b4ae0 con 0x7fc9080b4700 2024-03-25T19:35:29.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.055+0000 7fc90cd69640 10 monclient: _finish_auth 0 2024-03-25T19:35:29.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.055+0000 7fc90cd69640 10 monclient: _check_auth_tickets 2024-03-25T19:35:29.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.055+0000 7fc90cd69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:29.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.055+0000 7fc90cd69640 1 -- 172.21.15.111:0/2385940262 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fc900001690 con 0x7fc9080b4700 2024-03-25T19:35:29.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.055+0000 7fc90ffcf640 5 monclient: authenticate success, global_id 4121 2024-03-25T19:35:29.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.055+0000 7fc90ffcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:29.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.055+0000 7fc90cd69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:29.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.055+0000 7fc90cd69640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:29.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.055+0000 7fc90cd69640 10 monclient: dump: 2024-03-25T19:35:29.055 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:29.055 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:29.055 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:29.055 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:29.056 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:29.056 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:29.056 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:29.056 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:29.056 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:29.056 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:29.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.055+0000 7fc90ffcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:29.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.081+0000 7fc90cd69640 1 -- 172.21.15.111:0/2385940262 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9000017f0 con 0x7fc9080b4700 2024-03-25T19:35:29.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.081+0000 7fc90cd69640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:29.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.081+0000 7fc90cd69640 1 -- 172.21.15.111:0/2385940262 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fc900001c00 con 0x7fc9080b4700 2024-03-25T19:35:29.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.081+0000 7fc90cd69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:29.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.081+0000 7fc90cd69640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:29.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.081+0000 7fc90cd69640 10 monclient: dump: 2024-03-25T19:35:29.080 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:29.080 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:29.081 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:29.081 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:29.081 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:29.081 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:29.081 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:29.081 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:29.081 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:29.081 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:29.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.081+0000 7fc90ffcf640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:29.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.081+0000 7fc90ffcf640 4 set_mon_vals no callback set 2024-03-25T19:35:29.082 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.082+0000 7fc90ffcf640 10 monclient: shutdown 2024-03-25T19:35:29.082 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.082+0000 7fc90ffcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:29.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.082+0000 7fc90ffcf640 1 -- 172.21.15.111:0/2385940262 >> v1:172.21.15.44:6790/0 conn(0x7fc9080b4700 legacy=0x7fc908142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:29.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.082+0000 7fc90ffcf640 1 -- 172.21.15.111:0/2385940262 shutdown_connections 2024-03-25T19:35:29.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.082+0000 7fc90ffcf640 1 -- 172.21.15.111:0/2385940262 >> 172.21.15.111:0/2385940262 conn(0x7fc9080668e0 msgr2=0x7fc908066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:29.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.082+0000 7fc90ffcf640 1 -- 172.21.15.111:0/2385940262 shutdown_connections 2024-03-25T19:35:29.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.082+0000 7fc90ffcf640 1 -- 172.21.15.111:0/2385940262 wait complete. 2024-03-25T19:35:29.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.083+0000 7fc90ffcf640 10 monclient: build_initial_monmap 2024-03-25T19:35:29.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.083+0000 7fc90ffcf640 10 monclient: monmap: 2024-03-25T19:35:29.083 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T19:35:29.083 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:29.083 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 0.000000 2024-03-25T19:35:29.083 INFO:teuthology.orchestra.run.smithi111.stderr:created 0.000000 2024-03-25T19:35:29.083 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:29.083 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 1 2024-03-25T19:35:29.083 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:29.084 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:35:29.084 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:35:29.084 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:29.084 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:29.083+0000 7f882064b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:35:29.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.083+0000 7fc90ffcf640 1 librados: starting msgr at 2024-03-25T19:35:29.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.083+0000 7fc90ffcf640 1 librados: starting objecter 2024-03-25T19:35:29.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.083+0000 7fc90ffcf640 1 Processor -- start 2024-03-25T19:35:29.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.083+0000 7fc90ffcf640 1 -- start start 2024-03-25T19:35:29.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.083+0000 7fc90ffcf640 1 librados: setting wanted keys 2024-03-25T19:35:29.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.083+0000 7fc90ffcf640 1 librados: calling monclient init 2024-03-25T19:35:29.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.083+0000 7fc90ffcf640 10 monclient: init 2024-03-25T19:35:29.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.084+0000 7fc90ffcf640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:29.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.084+0000 7fc90ffcf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:35:29.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.084+0000 7fc90ffcf640 10 monclient(hunting): picked mon.noname-a con 0x7fc9080a96b0 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:29.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.084+0000 7fc90ffcf640 10 monclient(hunting): picked mon.noname-c con 0x7fc9080b4700 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:29.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.084+0000 7fc90ffcf640 10 monclient(hunting): picked mon.noname-b con 0x7fc908142a20 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:29.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.084+0000 7fc90ffcf640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9080adb20 con 0x7fc9080a96b0 2024-03-25T19:35:29.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.084+0000 7fc90ffcf640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9081349a0 con 0x7fc908142a20 2024-03-25T19:35:29.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.084+0000 7fc90ffcf640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9080acf80 con 0x7fc9080b4700 2024-03-25T19:35:29.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.084+0000 7fc90ffcf640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:29.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.084+0000 7fc90ffcf640 10 monclient(hunting): authenticate will time out at 5350.219057s 2024-03-25T19:35:29.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.084+0000 7fc90d56a640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fc9080b4700 0x7fc908133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.111:42390/0 (socket says 172.21.15.111:42390) 2024-03-25T19:35:29.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.084+0000 7fc90d56a640 1 -- 172.21.15.111:0/2200523649 learned_addr learned my addr 172.21.15.111:0/2200523649 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T19:35:29.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.085+0000 7fc8fe7fc640 10 client.?.objecter ms_handle_connect 0x7fc9080b4700 2024-03-25T19:35:29.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.085+0000 7fc8fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:29.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.085+0000 7fc8fe7fc640 10 client.?.objecter ms_handle_connect 0x7fc908142a20 2024-03-25T19:35:29.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.085+0000 7fc8fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:29.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.085+0000 7fc8fe7fc640 10 client.?.objecter ms_handle_connect 0x7fc9080a96b0 2024-03-25T19:35:29.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.085+0000 7fc8fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:29.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.086+0000 7fc8fe7fc640 1 -- 172.21.15.111:0/2200523649 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685869121 0 0) 0x7fc9080acf80 con 0x7fc9080b4700 2024-03-25T19:35:29.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.086+0000 7fc8fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:29.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.086+0000 7fc8fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:29.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.086+0000 7fc8fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:29.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.086+0000 7fc8fe7fc640 10 monclient(hunting): my global_id is 4124 2024-03-25T19:35:29.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.086+0000 7fc8fe7fc640 1 -- 172.21.15.111:0/2200523649 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8ec004eb0 con 0x7fc9080b4700 2024-03-25T19:35:29.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.086+0000 7fc8fe7fc640 1 -- 172.21.15.111:0/2200523649 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173609354 0 0) 0x7fc9081349a0 con 0x7fc908142a20 2024-03-25T19:35:29.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.086+0000 7fc8fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:29.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.086+0000 7fc8fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:29.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.086+0000 7fc8fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:29.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.086+0000 7fc8fe7fc640 10 monclient(hunting): my global_id is 4123 2024-03-25T19:35:29.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.086+0000 7fc8fe7fc640 1 -- 172.21.15.111:0/2200523649 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9080acf80 con 0x7fc908142a20 2024-03-25T19:35:29.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.087+0000 7fc8fe7fc640 1 -- 172.21.15.111:0/2200523649 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3636163388 0 0) 0x7fc9080adb20 con 0x7fc9080a96b0 2024-03-25T19:35:29.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.087+0000 7fc8fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:29.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.087+0000 7fc8fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:29.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.087+0000 7fc8fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:29.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.087+0000 7fc8fe7fc640 10 monclient(hunting): my global_id is 4125 2024-03-25T19:35:29.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.087+0000 7fc8fe7fc640 1 -- 172.21.15.111:0/2200523649 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9081349a0 con 0x7fc9080a96b0 2024-03-25T19:35:29.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.091+0000 7fc8fe7fc640 1 -- 172.21.15.111:0/2200523649 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4139163729 0 0) 0x7fc8ec004eb0 con 0x7fc9080b4700 2024-03-25T19:35:29.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.091+0000 7fc8fe7fc640 1 -- 172.21.15.111:0/2200523649 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9080adb20 con 0x7fc9080b4700 2024-03-25T19:35:29.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.091+0000 7fc8fe7fc640 1 -- 172.21.15.111:0/2200523649 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fc8f8001a10 con 0x7fc9080b4700 2024-03-25T19:35:29.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.091+0000 7fc8fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:29.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.091+0000 7fc8fe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:29.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.091+0000 7fc8fe7fc640 10 monclient(hunting): dump: 2024-03-25T19:35:29.091 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:29.091 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:29.091 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:29.091 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:29.091 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:29.091 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:29.091 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:29.092 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:29.092 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:29.092 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:29.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.091+0000 7fc8fe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:29.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.091+0000 7fc8fe7fc640 1 -- 172.21.15.111:0/2200523649 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3058604579 0 0) 0x7fc9080acf80 con 0x7fc908142a20 2024-03-25T19:35:29.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.091+0000 7fc8fe7fc640 1 -- 172.21.15.111:0/2200523649 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8ec004eb0 con 0x7fc908142a20 2024-03-25T19:35:29.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.091+0000 7fc8fe7fc640 1 -- 172.21.15.111:0/2200523649 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fc904001b50 con 0x7fc908142a20 2024-03-25T19:35:29.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.091+0000 7fc8fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:29.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.091+0000 7fc8fe7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:29.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.091+0000 7fc8fe7fc640 10 monclient(hunting): dump: 2024-03-25T19:35:29.092 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:29.092 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:29.092 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:29.092 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:29.092 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:29.092 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:29.092 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:29.093 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:29.093 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:29.093 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:29.093 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T19:35:29.093+0000 7f54ee4c7040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:35:29.094 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.093+0000 7fc8fe7fc640 1 -- 172.21.15.111:0/2200523649 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2774466529 0 0) 0x7fc9081349a0 con 0x7fc9080a96b0 2024-03-25T19:35:29.094 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.093+0000 7fc8fe7fc640 1 -- 172.21.15.111:0/2200523649 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9080acf80 con 0x7fc9080a96b0 2024-03-25T19:35:29.094 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.093+0000 7fc8fe7fc640 1 -- 172.21.15.111:0/2200523649 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fc900001b00 con 0x7fc9080a96b0 2024-03-25T19:35:29.094 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.093+0000 7fc8fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:29.094 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.093+0000 7fc8fe7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:29.094 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.093+0000 7fc8fe7fc640 10 monclient(hunting): dump: 2024-03-25T19:35:29.094 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:29.094 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:29.094 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:29.094 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:29.094 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:29.094 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:29.095 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:29.095 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:29.095 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:29.095 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:29.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.097+0000 7fc8fe7fc640 1 -- 172.21.15.111:0/2200523649 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 603318324 0 0) 0x7fc9080acf80 con 0x7fc9080a96b0 2024-03-25T19:35:29.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.097+0000 7fc8fe7fc640 1 -- 172.21.15.111:0/2200523649 >> v1:172.21.15.44:6790/0 conn(0x7fc9080b4700 legacy=0x7fc908133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:29.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.097+0000 7fc8fe7fc640 1 -- 172.21.15.111:0/2200523649 >> v1:172.21.15.111:6789/0 conn(0x7fc908142a20 legacy=0x7fc908134290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:29.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.097+0000 7fc8fe7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:35:29.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.097+0000 7fc8fe7fc640 1 monclient: found mon.a 2024-03-25T19:35:29.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.097+0000 7fc8fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:29.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.097+0000 7fc8fe7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:29.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.097+0000 7fc8fe7fc640 1 -- 172.21.15.111:0/2200523649 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9080ade00 con 0x7fc9080a96b0 2024-03-25T19:35:29.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.097+0000 7fc8fe7fc640 10 monclient: _finish_auth 0 2024-03-25T19:35:29.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.097+0000 7fc8fe7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:35:29.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.097+0000 7fc8fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:29.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.097+0000 7fc90ffcf640 5 monclient: authenticate success, global_id 4125 2024-03-25T19:35:29.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.097+0000 7fc90ffcf640 10 monclient: _renew_subs 2024-03-25T19:35:29.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.097+0000 7fc90ffcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:29.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.097+0000 7fc90ffcf640 1 -- 172.21.15.111:0/2200523649 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9080aca60 con 0x7fc9080a96b0 2024-03-25T19:35:29.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.097+0000 7fc90ffcf640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:29.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.097+0000 7fc90ffcf640 10 monclient: _renew_subs 2024-03-25T19:35:29.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.097+0000 7fc90ffcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:29.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.097+0000 7fc90ffcf640 1 -- 172.21.15.111:0/2200523649 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc908139f90 con 0x7fc9080a96b0 2024-03-25T19:35:29.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.097+0000 7fc90ffcf640 1 librados: init done 2024-03-25T19:35:29.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.098+0000 7fc90ffcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:29.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.098+0000 7fc90ffcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:29.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.098+0000 7fc90ffcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:29.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.098+0000 7fc90ffcf640 1 -- 172.21.15.111:0/2200523649 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc90804f730 con 0x7fc9080a96b0 2024-03-25T19:35:29.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.101+0000 7fc8fe7fc640 1 -- 172.21.15.111:0/2200523649 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc900001c60 con 0x7fc9080a96b0 2024-03-25T19:35:29.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.101+0000 7fc8fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:29.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.101+0000 7fc8fe7fc640 1 -- 172.21.15.111:0/2200523649 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fc900004c30 con 0x7fc9080a96b0 2024-03-25T19:35:29.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.101+0000 7fc8fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:29.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.101+0000 7fc8fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:29.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.101+0000 7fc8fe7fc640 10 monclient: dump: 2024-03-25T19:35:29.101 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:29.101 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:29.101 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:29.101 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:29.101 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:29.101 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:29.101 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:29.101 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:29.101 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:29.101 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:29.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.101+0000 7fc8fe7fc640 1 -- 172.21.15.111:0/2200523649 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc900005120 con 0x7fc9080a96b0 2024-03-25T19:35:29.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.101+0000 7fc8fffff640 4 set_mon_vals no callback set 2024-03-25T19:35:29.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.101+0000 7fc8fe7fc640 1 -- 172.21.15.111:0/2200523649 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3765387119 0 0) 0x7fc900006660 con 0x7fc9080a96b0 2024-03-25T19:35:29.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.101+0000 7fc8fe7fc640 10 client.4125.objecter ms_dispatch 0x7fc908000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T19:35:29.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.101+0000 7fc8fe7fc640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T19:35:29.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.101+0000 7fc8fe7fc640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T19:35:29.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.101+0000 7fc8fe7fc640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T19:35:29.137 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:29.138+0000 7f882064b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:35:29.168 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T19:35:29.168+0000 7f54ee4c7040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:35:29.183 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:29.184+0000 7f882064b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:35:29.201 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.202+0000 7fc8fe7fc640 1 -- 172.21.15.111:0/2200523649 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc900006960 con 0x7fc9080a96b0 2024-03-25T19:35:29.201 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.203+0000 7fc8fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:29.201 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.203+0000 7fc8fe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:29.216 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T19:35:29.216+0000 7f54ee4c7040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:35:29.271 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T19:35:29.272+0000 7f54ee4c7040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:35:29.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.278+0000 7fc90ffcf640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "24b49d5b-ed7c-4388-a11d-0c292ef38dbd", "id": 1}] 2024-03-25T19:35:29.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.278+0000 7fc90ffcf640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "24b49d5b-ed7c-4388-a11d-0c292ef38dbd", "id": 1}] 2024-03-25T19:35:29.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.278+0000 7fc90ffcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:29.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.278+0000 7fc90ffcf640 1 -- 172.21.15.111:0/2200523649 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd new", "uuid": "24b49d5b-ed7c-4388-a11d-0c292ef38dbd", "id": 1} v 0) v1 -- 0x7fc9080b7600 con 0x7fc9080a96b0 2024-03-25T19:35:29.326 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T19:35:29.326+0000 7f54ee4c7040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:35:29.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.334+0000 7fc8fe7fc640 1 -- 172.21.15.111:0/2200523649 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "24b49d5b-ed7c-4388-a11d-0c292ef38dbd", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3140626919 0 940483276) 0x7fc900006b80 con 0x7fc9080a96b0 2024-03-25T19:35:29.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.334+0000 7fc8fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "24b49d5b-ed7c-4388-a11d-0c292ef38dbd", "id": 1}] 2024-03-25T19:35:29.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.334+0000 7fc8fe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:29.333 INFO:teuthology.orchestra.run.smithi111.stdout:1 2024-03-25T19:35:29.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.335+0000 7fc90ffcf640 10 librados: watch_flush enter 2024-03-25T19:35:29.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.335+0000 7fc90ffcf640 10 librados: watch_flush exit 2024-03-25T19:35:29.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.335+0000 7fc90ffcf640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T19:35:29.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.335+0000 7fc90ffcf640 10 client.4125.objecter successfully canceled tick 2024-03-25T19:35:29.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.335+0000 7fc90ffcf640 10 monclient: shutdown 2024-03-25T19:35:29.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.335+0000 7fc90ffcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:29.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.335+0000 7fc90ffcf640 1 -- 172.21.15.111:0/2200523649 >> v1:172.21.15.44:6789/0 conn(0x7fc9080a96b0 legacy=0x7fc9081396a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:29.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.335+0000 7fc90ffcf640 1 -- 172.21.15.111:0/2200523649 shutdown_connections 2024-03-25T19:35:29.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.335+0000 7fc90ffcf640 1 -- 172.21.15.111:0/2200523649 >> 172.21.15.111:0/2200523649 conn(0x7fc9080668e0 msgr2=0x7fc9080b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:29.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.335+0000 7fc90ffcf640 1 -- 172.21.15.111:0/2200523649 shutdown_connections 2024-03-25T19:35:29.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.335+0000 7fc90ffcf640 1 -- 172.21.15.111:0/2200523649 wait complete. 2024-03-25T19:35:29.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.335+0000 7fc90ffcf640 1 librados: shutdown 2024-03-25T19:35:29.344 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new b6f34510-1d0d-46d1-8e66-17fd9dd62e77 2 2024-03-25T19:35:29.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.486+0000 7f12319af640 10 monclient: get_monmap_and_config 2024-03-25T19:35:29.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.486+0000 7f12319af640 10 monclient: build_initial_monmap 2024-03-25T19:35:29.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.486+0000 7f12319af640 10 monclient: monmap: 2024-03-25T19:35:29.485 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T19:35:29.485 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:29.485 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:35:29.487028+0000 2024-03-25T19:35:29.485 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:35:29.487028+0000 2024-03-25T19:35:29.486 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:29.486 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:29.486 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:29.486 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:35:29.486 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:35:29.486 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:29.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.487+0000 7f12319af640 1 Processor -- start 2024-03-25T19:35:29.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.487+0000 7f12319af640 1 -- start start 2024-03-25T19:35:29.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.487+0000 7f12319af640 10 monclient: init 2024-03-25T19:35:29.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.488+0000 7f12319af640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:29.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.488+0000 7f12319af640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:35:29.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.488+0000 7f12319af640 10 monclient(hunting): picked mon.noname-a con 0x7f122c0b4700 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:29.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.488+0000 7f12319af640 10 monclient(hunting): picked mon.noname-b con 0x7f122c142a20 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:29.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.488+0000 7f12319af640 10 monclient(hunting): picked mon.noname-c con 0x7f122c0a96b0 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:29.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.488+0000 7f12319af640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f122c0a9d60 con 0x7f122c0b4700 2024-03-25T19:35:29.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.488+0000 7f12319af640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f122c13ff10 con 0x7f122c0a96b0 2024-03-25T19:35:29.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.488+0000 7f12319af640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f122c0aa1f0 con 0x7f122c142a20 2024-03-25T19:35:29.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.488+0000 7f12319af640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:29.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.488+0000 7f12319af640 10 monclient(hunting): authenticate will time out at 5350.623050s 2024-03-25T19:35:29.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.488+0000 7f1223fff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f122c142a20 0x7f122c13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.111:42404/0 (socket says 172.21.15.111:42404) 2024-03-25T19:35:29.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.488+0000 7f1223fff640 1 -- 172.21.15.111:0/3548707685 learned_addr learned my addr 172.21.15.111:0/3548707685 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T19:35:29.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.491+0000 7f122a7fc640 1 -- 172.21.15.111:0/3548707685 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249357164 0 0) 0x7f122c0aa1f0 con 0x7f122c142a20 2024-03-25T19:35:29.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.491+0000 7f122a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:29.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.491+0000 7f122a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:29.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.491+0000 7f122a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:29.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.491+0000 7f122a7fc640 10 monclient(hunting): my global_id is 4127 2024-03-25T19:35:29.491 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.491+0000 7f122a7fc640 1 -- 172.21.15.111:0/3548707685 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1218003520 con 0x7f122c142a20 2024-03-25T19:35:29.491 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.491+0000 7f122a7fc640 1 -- 172.21.15.111:0/3548707685 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927958885 0 0) 0x7f122c13ff10 con 0x7f122c0a96b0 2024-03-25T19:35:29.491 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.491+0000 7f122a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:29.491 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.491+0000 7f122a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:29.491 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.491+0000 7f122a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:29.491 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.491+0000 7f122a7fc640 10 monclient(hunting): my global_id is 4126 2024-03-25T19:35:29.491 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.491+0000 7f122a7fc640 1 -- 172.21.15.111:0/3548707685 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f122c0aa1f0 con 0x7f122c0a96b0 2024-03-25T19:35:29.491 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.491+0000 7f122a7fc640 1 -- 172.21.15.111:0/3548707685 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 511302263 0 0) 0x7f122c0a9d60 con 0x7f122c0b4700 2024-03-25T19:35:29.491 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.491+0000 7f122a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:29.491 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.491+0000 7f122a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:29.491 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.491+0000 7f122a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:29.491 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.491+0000 7f122a7fc640 10 monclient(hunting): my global_id is 4128 2024-03-25T19:35:29.491 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.491+0000 7f122a7fc640 1 -- 172.21.15.111:0/3548707685 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f122c13ff10 con 0x7f122c0b4700 2024-03-25T19:35:29.493 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.494+0000 7f122a7fc640 1 -- 172.21.15.111:0/3548707685 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4226755569 0 0) 0x7f1218003520 con 0x7f122c142a20 2024-03-25T19:35:29.493 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.494+0000 7f122a7fc640 1 -- 172.21.15.111:0/3548707685 >> v1:172.21.15.111:6789/0 conn(0x7f122c0a96b0 legacy=0x7f122c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:29.493 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.494+0000 7f122a7fc640 1 -- 172.21.15.111:0/3548707685 >> v1:172.21.15.44:6789/0 conn(0x7f122c0b4700 legacy=0x7f122c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:29.493 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.494+0000 7f122a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:35:29.493 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.494+0000 7f122a7fc640 1 monclient: found mon.noname-b 2024-03-25T19:35:29.493 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.494+0000 7f122a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:29.493 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.494+0000 7f122a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:35:29.493 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.494+0000 7f122a7fc640 1 -- 172.21.15.111:0/3548707685 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f122c0b4ae0 con 0x7f122c142a20 2024-03-25T19:35:29.493 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.494+0000 7f122a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:35:29.493 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.494+0000 7f122a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:35:29.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.494+0000 7f122a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:29.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.494+0000 7f122a7fc640 1 -- 172.21.15.111:0/3548707685 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f1214001690 con 0x7f122c142a20 2024-03-25T19:35:29.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.494+0000 7f12319af640 5 monclient: authenticate success, global_id 4127 2024-03-25T19:35:29.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.494+0000 7f12319af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:29.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.494+0000 7f122a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:29.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.494+0000 7f122a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:29.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.494+0000 7f122a7fc640 10 monclient: dump: 2024-03-25T19:35:29.494 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:29.494 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:29.494 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:29.494 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:29.494 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:29.494 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:29.494 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:29.494 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:29.495 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:29.495 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:29.495 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.494+0000 7f12319af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:29.495 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.497+0000 7f122a7fc640 1 -- 172.21.15.111:0/3548707685 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12140017f0 con 0x7f122c142a20 2024-03-25T19:35:29.495 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.497+0000 7f122a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:29.495 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.497+0000 7f122a7fc640 1 -- 172.21.15.111:0/3548707685 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f1214001c00 con 0x7f122c142a20 2024-03-25T19:35:29.496 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.497+0000 7f122a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:29.496 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.497+0000 7f122a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:29.496 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.497+0000 7f122a7fc640 10 monclient: dump: 2024-03-25T19:35:29.496 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:29.496 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:29.496 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:29.496 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:29.496 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:29.496 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:29.496 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:29.496 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:29.496 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:29.496 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:29.496 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.497+0000 7f12319af640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:29.496 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.497+0000 7f12319af640 4 set_mon_vals no callback set 2024-03-25T19:35:29.499 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.497+0000 7f12319af640 10 monclient: shutdown 2024-03-25T19:35:29.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.497+0000 7f12319af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:29.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.497+0000 7f12319af640 1 -- 172.21.15.111:0/3548707685 >> v1:172.21.15.44:6790/0 conn(0x7f122c142a20 legacy=0x7f122c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:29.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.497+0000 7f12319af640 1 -- 172.21.15.111:0/3548707685 shutdown_connections 2024-03-25T19:35:29.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.497+0000 7f12319af640 1 -- 172.21.15.111:0/3548707685 >> 172.21.15.111:0/3548707685 conn(0x7f122c0668e0 msgr2=0x7f122c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:29.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.497+0000 7f12319af640 1 -- 172.21.15.111:0/3548707685 shutdown_connections 2024-03-25T19:35:29.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.497+0000 7f12319af640 1 -- 172.21.15.111:0/3548707685 wait complete. 2024-03-25T19:35:29.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.498+0000 7f12319af640 10 monclient: build_initial_monmap 2024-03-25T19:35:29.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.498+0000 7f12319af640 10 monclient: monmap: 2024-03-25T19:35:29.502 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T19:35:29.502 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:29.502 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 0.000000 2024-03-25T19:35:29.502 INFO:teuthology.orchestra.run.smithi111.stderr:created 0.000000 2024-03-25T19:35:29.502 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:29.502 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 1 2024-03-25T19:35:29.502 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:29.502 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:35:29.502 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:35:29.502 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:29.502 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.498+0000 7f12319af640 1 librados: starting msgr at 2024-03-25T19:35:29.502 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.498+0000 7f12319af640 1 librados: starting objecter 2024-03-25T19:35:29.502 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.498+0000 7f12319af640 1 Processor -- start 2024-03-25T19:35:29.502 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.498+0000 7f12319af640 1 -- start start 2024-03-25T19:35:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.498+0000 7f12319af640 1 librados: setting wanted keys 2024-03-25T19:35:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.498+0000 7f12319af640 1 librados: calling monclient init 2024-03-25T19:35:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.498+0000 7f12319af640 10 monclient: init 2024-03-25T19:35:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.499+0000 7f12319af640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.499+0000 7f12319af640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.499+0000 7f12319af640 10 monclient(hunting): picked mon.noname-b con 0x7f122c0a96b0 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.499+0000 7f12319af640 10 monclient(hunting): picked mon.noname-a con 0x7f122c0b4700 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.499+0000 7f12319af640 10 monclient(hunting): picked mon.noname-c con 0x7f122c142a20 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.499+0000 7f12319af640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f122c1dab30 con 0x7f122c0b4700 2024-03-25T19:35:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.499+0000 7f12319af640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f122c0ab880 con 0x7f122c0a96b0 2024-03-25T19:35:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.499+0000 7f12319af640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f122c1d7510 con 0x7f122c142a20 2024-03-25T19:35:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.499+0000 7f12319af640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.499+0000 7f12319af640 10 monclient(hunting): authenticate will time out at 5350.634230s 2024-03-25T19:35:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.499+0000 7f122affd640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f122c0a96b0 0x7f122c1d7b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:47260/0 (socket says 172.21.15.111:47260) 2024-03-25T19:35:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.499+0000 7f122affd640 1 -- 172.21.15.111:0/834849176 learned_addr learned my addr 172.21.15.111:0/834849176 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T19:35:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.500+0000 7f1228ff9640 10 client.?.objecter ms_handle_connect 0x7f122c142a20 2024-03-25T19:35:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.500+0000 7f1228ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.500+0000 7f1228ff9640 10 client.?.objecter ms_handle_connect 0x7f122c0b4700 2024-03-25T19:35:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.500+0000 7f1228ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.500+0000 7f1228ff9640 10 client.?.objecter ms_handle_connect 0x7f122c0a96b0 2024-03-25T19:35:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.500+0000 7f1228ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.502+0000 7f1228ff9640 1 -- 172.21.15.111:0/834849176 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191392737 0 0) 0x7f122c1d7510 con 0x7f122c142a20 2024-03-25T19:35:29.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.502+0000 7f1228ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:29.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.502+0000 7f1228ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:29.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.502+0000 7f1228ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:29.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.502+0000 7f1228ff9640 10 monclient(hunting): my global_id is 4130 2024-03-25T19:35:29.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.502+0000 7f1228ff9640 1 -- 172.21.15.111:0/834849176 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f120c004eb0 con 0x7f122c142a20 2024-03-25T19:35:29.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.502+0000 7f1228ff9640 1 -- 172.21.15.111:0/834849176 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859728075 0 0) 0x7f122c1dab30 con 0x7f122c0b4700 2024-03-25T19:35:29.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.502+0000 7f1228ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:29.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.502+0000 7f1228ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:29.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.502+0000 7f1228ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:29.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.502+0000 7f1228ff9640 10 monclient(hunting): my global_id is 4131 2024-03-25T19:35:29.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.502+0000 7f1228ff9640 1 -- 172.21.15.111:0/834849176 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f122c1d7510 con 0x7f122c0b4700 2024-03-25T19:35:29.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.502+0000 7f1228ff9640 1 -- 172.21.15.111:0/834849176 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157353166 0 0) 0x7f122c0ab880 con 0x7f122c0a96b0 2024-03-25T19:35:29.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.502+0000 7f1228ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:29.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.502+0000 7f1228ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:29.506 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.502+0000 7f1228ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:29.506 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.502+0000 7f1228ff9640 10 monclient(hunting): my global_id is 4129 2024-03-25T19:35:29.506 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.502+0000 7f1228ff9640 1 -- 172.21.15.111:0/834849176 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f122c1dab30 con 0x7f122c0a96b0 2024-03-25T19:35:29.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.506+0000 7f1228ff9640 1 -- 172.21.15.111:0/834849176 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2972188854 0 0) 0x7f122c1dab30 con 0x7f122c0a96b0 2024-03-25T19:35:29.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.506+0000 7f1228ff9640 1 -- 172.21.15.111:0/834849176 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f122c0ab880 con 0x7f122c0a96b0 2024-03-25T19:35:29.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.506+0000 7f1228ff9640 1 -- 172.21.15.111:0/834849176 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f121c0019b0 con 0x7f122c0a96b0 2024-03-25T19:35:29.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.506+0000 7f1228ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:29.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.506+0000 7f1228ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:29.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.506+0000 7f1228ff9640 10 monclient(hunting): dump: 2024-03-25T19:35:29.507 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:29.507 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:29.508 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:29.508 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:29.508 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:29.508 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:29.508 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:29.508 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:29.508 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:29.508 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:29.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.506+0000 7f1228ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:29.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.506+0000 7f1228ff9640 1 -- 172.21.15.111:0/834849176 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2122911785 0 0) 0x7f120c004eb0 con 0x7f122c142a20 2024-03-25T19:35:29.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.506+0000 7f1228ff9640 1 -- 172.21.15.111:0/834849176 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f122c1dab30 con 0x7f122c142a20 2024-03-25T19:35:29.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.506+0000 7f1228ff9640 1 -- 172.21.15.111:0/834849176 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f1224001bd0 con 0x7f122c142a20 2024-03-25T19:35:29.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.506+0000 7f1228ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:29.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.506+0000 7f1228ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:29.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.507+0000 7f1228ff9640 10 monclient(hunting): dump: 2024-03-25T19:35:29.509 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:29.509 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:29.509 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:29.509 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:29.509 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:29.509 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:29.509 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:29.509 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:29.509 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:29.509 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:29.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.508+0000 7f1228ff9640 1 -- 172.21.15.111:0/834849176 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3160003415 0 0) 0x7f122c1d7510 con 0x7f122c0b4700 2024-03-25T19:35:29.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.508+0000 7f1228ff9640 1 -- 172.21.15.111:0/834849176 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f120c004eb0 con 0x7f122c0b4700 2024-03-25T19:35:29.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.508+0000 7f1228ff9640 1 -- 172.21.15.111:0/834849176 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f1214001b00 con 0x7f122c0b4700 2024-03-25T19:35:29.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.508+0000 7f1228ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:29.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.508+0000 7f1228ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:29.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.508+0000 7f1228ff9640 10 monclient(hunting): dump: 2024-03-25T19:35:29.510 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:29.510 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:29.510 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:29.510 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:29.510 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:29.510 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:29.510 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:29.510 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:29.510 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:29.510 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.510+0000 7f1228ff9640 1 -- 172.21.15.111:0/834849176 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4184569694 0 0) 0x7f122c0ab880 con 0x7f122c0a96b0 2024-03-25T19:35:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.510+0000 7f1228ff9640 1 -- 172.21.15.111:0/834849176 >> v1:172.21.15.44:6790/0 conn(0x7f122c142a20 legacy=0x7f122c0ac340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.510+0000 7f1228ff9640 1 -- 172.21.15.111:0/834849176 >> v1:172.21.15.44:6789/0 conn(0x7f122c0b4700 legacy=0x7f122c1db050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.510+0000 7f1228ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:35:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.510+0000 7f1228ff9640 1 monclient: found mon.b 2024-03-25T19:35:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.510+0000 7f1228ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.510+0000 7f1228ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.510+0000 7f1228ff9640 1 -- 172.21.15.111:0/834849176 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f122c1d76a0 con 0x7f122c0a96b0 2024-03-25T19:35:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.510+0000 7f1228ff9640 10 monclient: _finish_auth 0 2024-03-25T19:35:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.510+0000 7f1228ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:35:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.510+0000 7f1228ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.510+0000 7f12319af640 5 monclient: authenticate success, global_id 4129 2024-03-25T19:35:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.510+0000 7f12319af640 10 monclient: _renew_subs 2024-03-25T19:35:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.510+0000 7f12319af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:29.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.510+0000 7f12319af640 1 -- 172.21.15.111:0/834849176 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f122c1d8280 con 0x7f122c0a96b0 2024-03-25T19:35:29.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.510+0000 7f12319af640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:29.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.510+0000 7f12319af640 10 monclient: _renew_subs 2024-03-25T19:35:29.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.510+0000 7f12319af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:29.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.510+0000 7f12319af640 1 -- 172.21.15.111:0/834849176 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f122c1db940 con 0x7f122c0a96b0 2024-03-25T19:35:29.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.510+0000 7f12319af640 1 librados: init done 2024-03-25T19:35:29.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.511+0000 7f12319af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:29.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.511+0000 7f12319af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:29.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.511+0000 7f12319af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:29.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.511+0000 7f12319af640 1 -- 172.21.15.111:0/834849176 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f122c0ae0a0 con 0x7f122c0a96b0 2024-03-25T19:35:29.583 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:29.584+0000 7f882064b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:35:29.722 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:29.723+0000 7f882064b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:35:29.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.730+0000 7f1228ff9640 1 -- 172.21.15.111:0/834849176 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f121c003610 con 0x7f122c0a96b0 2024-03-25T19:35:29.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.730+0000 7f1228ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:29.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.730+0000 7f1228ff9640 1 -- 172.21.15.111:0/834849176 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f121c003920 con 0x7f122c0a96b0 2024-03-25T19:35:29.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.730+0000 7f1228ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:29.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.730+0000 7f1228ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:29.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.730+0000 7f1228ff9640 10 monclient: dump: 2024-03-25T19:35:29.729 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:29.730 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:29.730 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:29.730 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:29.730 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:29.730 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:29.730 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:29.730 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:29.730 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:29.730 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:29.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.730+0000 7f1228ff9640 1 -- 172.21.15.111:0/834849176 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f121c003ed0 con 0x7f122c0a96b0 2024-03-25T19:35:29.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.730+0000 7f1229ffb640 4 set_mon_vals no callback set 2024-03-25T19:35:29.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.730+0000 7f1228ff9640 1 -- 172.21.15.111:0/834849176 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 128643867 0 0) 0x7f121c0065b0 con 0x7f122c0a96b0 2024-03-25T19:35:29.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.730+0000 7f1228ff9640 10 client.4129.objecter ms_dispatch 0x7f122c000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T19:35:29.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.730+0000 7f1228ff9640 3 client.4129.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T19:35:29.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.730+0000 7f1228ff9640 3 client.4129.objecter handle_osd_map decoding full epoch 4 2024-03-25T19:35:29.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.730+0000 7f1228ff9640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-25T19:35:29.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.730+0000 7f1228ff9640 1 -- 172.21.15.111:0/834849176 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f121c006990 con 0x7f122c0a96b0 2024-03-25T19:35:29.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.730+0000 7f1228ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:29.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.730+0000 7f1228ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:29.731 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T19:35:29.730+0000 7f54ee4c7040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:35:29.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.827+0000 7f12319af640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b6f34510-1d0d-46d1-8e66-17fd9dd62e77", "id": 2}] 2024-03-25T19:35:29.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.827+0000 7f12319af640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b6f34510-1d0d-46d1-8e66-17fd9dd62e77", "id": 2}] 2024-03-25T19:35:29.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.827+0000 7f12319af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:29.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.827+0000 7f12319af640 1 -- 172.21.15.111:0/834849176 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b6f34510-1d0d-46d1-8e66-17fd9dd62e77", "id": 2} v 0) v1 -- 0x7f122c04f730 con 0x7f122c0a96b0 2024-03-25T19:35:29.834 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:29.836+0000 7f882064b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:35:29.877 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T19:35:29.877+0000 7f54ee4c7040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:35:29.882 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:29.883+0000 7f882064b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:35:29.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.937+0000 7f1228ff9640 1 -- 172.21.15.111:0/834849176 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b6f34510-1d0d-46d1-8e66-17fd9dd62e77", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 554360069 0 216599125) 0x7f121c0043b0 con 0x7f122c0a96b0 2024-03-25T19:35:29.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.937+0000 7f1228ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b6f34510-1d0d-46d1-8e66-17fd9dd62e77", "id": 2}] 2024-03-25T19:35:29.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.937+0000 7f1228ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:29.936 INFO:teuthology.orchestra.run.smithi111.stdout:2 2024-03-25T19:35:29.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.938+0000 7f12319af640 10 librados: watch_flush enter 2024-03-25T19:35:29.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.938+0000 7f12319af640 10 librados: watch_flush exit 2024-03-25T19:35:29.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.938+0000 7f12319af640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-25T19:35:29.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.938+0000 7f12319af640 10 client.4129.objecter successfully canceled tick 2024-03-25T19:35:29.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.938+0000 7f12319af640 10 monclient: shutdown 2024-03-25T19:35:29.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.938+0000 7f12319af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:29.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.938+0000 7f12319af640 1 -- 172.21.15.111:0/834849176 >> v1:172.21.15.111:6789/0 conn(0x7f122c0a96b0 legacy=0x7f122c1d7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:29.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.938+0000 7f12319af640 1 -- 172.21.15.111:0/834849176 shutdown_connections 2024-03-25T19:35:29.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.938+0000 7f12319af640 1 -- 172.21.15.111:0/834849176 >> 172.21.15.111:0/834849176 conn(0x7f122c0668e0 msgr2=0x7f122c066cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:29.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.938+0000 7f12319af640 1 -- 172.21.15.111:0/834849176 shutdown_connections 2024-03-25T19:35:29.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.938+0000 7f12319af640 1 -- 172.21.15.111:0/834849176 wait complete. 2024-03-25T19:35:29.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:29.938+0000 7f12319af640 1 librados: shutdown 2024-03-25T19:35:29.947 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new c938187c-9648-474d-896c-4e91867dfc72 3 2024-03-25T19:35:30.001 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:30.002+0000 7f882064b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:35:30.019 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T19:35:30.019+0000 7f54ee4c7040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:35:30.045 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.046+0000 7fe64c315640 10 monclient: get_monmap_and_config 2024-03-25T19:35:30.045 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.046+0000 7fe64c315640 10 monclient: build_initial_monmap 2024-03-25T19:35:30.045 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.046+0000 7fe64c315640 10 monclient: monmap: 2024-03-25T19:35:30.045 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T19:35:30.045 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:30.045 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:35:30.046811+0000 2024-03-25T19:35:30.045 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:35:30.046811+0000 2024-03-25T19:35:30.045 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:30.045 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:30.046 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:30.046 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:35:30.046 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:35:30.046 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:30.047 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.047+0000 7fe64c315640 1 Processor -- start 2024-03-25T19:35:30.047 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.047+0000 7fe64c315640 1 -- start start 2024-03-25T19:35:30.047 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.047+0000 7fe64c315640 10 monclient: init 2024-03-25T19:35:30.047 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.047+0000 7fe64c315640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:30.047 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.047+0000 7fe64c315640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:30.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.048+0000 7fe64c315640 10 monclient(hunting): picked mon.noname-c con 0x7fe6440b4700 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:30.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.048+0000 7fe64c315640 10 monclient(hunting): picked mon.noname-a con 0x7fe644142a20 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:30.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.048+0000 7fe64c315640 10 monclient(hunting): picked mon.noname-b con 0x7fe6440a96b0 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:30.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.048+0000 7fe64c315640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6440a9d60 con 0x7fe644142a20 2024-03-25T19:35:30.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.048+0000 7fe64c315640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe64413ff10 con 0x7fe6440b4700 2024-03-25T19:35:30.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.048+0000 7fe64c315640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6440aa1f0 con 0x7fe6440a96b0 2024-03-25T19:35:30.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.048+0000 7fe64c315640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:30.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.048+0000 7fe64c315640 10 monclient(hunting): authenticate will time out at 5351.182830s 2024-03-25T19:35:30.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.048+0000 7fe6498b0640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fe644142a20 0x7fe64413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.111:44284/0 (socket says 172.21.15.111:44284) 2024-03-25T19:35:30.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.048+0000 7fe64a8b2640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fe6440a96b0 0x7fe64413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.111:42434/0 (socket says 172.21.15.111:42434) 2024-03-25T19:35:30.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.048+0000 7fe6498b0640 1 -- 172.21.15.111:0/639829879 learned_addr learned my addr 172.21.15.111:0/639829879 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T19:35:30.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.051+0000 7fe6490af640 1 -- 172.21.15.111:0/639829879 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800396339 0 0) 0x7fe6440aa1f0 con 0x7fe6440a96b0 2024-03-25T19:35:30.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.051+0000 7fe6490af640 10 monclient(hunting): _negotiate 2024-03-25T19:35:30.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.051+0000 7fe6490af640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:30.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.051+0000 7fe6490af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:30.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.051+0000 7fe6490af640 10 monclient(hunting): my global_id is 4133 2024-03-25T19:35:30.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.051+0000 7fe6490af640 1 -- 172.21.15.111:0/639829879 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe630003520 con 0x7fe6440a96b0 2024-03-25T19:35:30.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.051+0000 7fe6490af640 1 -- 172.21.15.111:0/639829879 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838454577 0 0) 0x7fe64413ff10 con 0x7fe6440b4700 2024-03-25T19:35:30.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.051+0000 7fe6490af640 10 monclient(hunting): _negotiate 2024-03-25T19:35:30.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.051+0000 7fe6490af640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:30.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.051+0000 7fe6490af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:30.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.051+0000 7fe6490af640 10 monclient(hunting): my global_id is 4132 2024-03-25T19:35:30.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.051+0000 7fe6490af640 1 -- 172.21.15.111:0/639829879 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6440aa1f0 con 0x7fe6440b4700 2024-03-25T19:35:30.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.051+0000 7fe6490af640 1 -- 172.21.15.111:0/639829879 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639738308 0 0) 0x7fe6440a9d60 con 0x7fe644142a20 2024-03-25T19:35:30.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.051+0000 7fe6490af640 10 monclient(hunting): _negotiate 2024-03-25T19:35:30.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.051+0000 7fe6490af640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:30.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.051+0000 7fe6490af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:30.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.051+0000 7fe6490af640 10 monclient(hunting): my global_id is 4134 2024-03-25T19:35:30.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.051+0000 7fe6490af640 1 -- 172.21.15.111:0/639829879 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe64413ff10 con 0x7fe644142a20 2024-03-25T19:35:30.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.054+0000 7fe6490af640 1 -- 172.21.15.111:0/639829879 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2252792743 0 0) 0x7fe630003520 con 0x7fe6440a96b0 2024-03-25T19:35:30.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.054+0000 7fe6490af640 1 -- 172.21.15.111:0/639829879 >> v1:172.21.15.111:6789/0 conn(0x7fe6440b4700 legacy=0x7fe644142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:30.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.054+0000 7fe6490af640 1 -- 172.21.15.111:0/639829879 >> v1:172.21.15.44:6789/0 conn(0x7fe644142a20 legacy=0x7fe64413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:30.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.054+0000 7fe6490af640 10 monclient: _finish_hunting 0 2024-03-25T19:35:30.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.054+0000 7fe6490af640 1 monclient: found mon.noname-b 2024-03-25T19:35:30.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.054+0000 7fe6490af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:30.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.054+0000 7fe6490af640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:35:30.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.054+0000 7fe6490af640 1 -- 172.21.15.111:0/639829879 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6440b4ae0 con 0x7fe6440a96b0 2024-03-25T19:35:30.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.054+0000 7fe6490af640 10 monclient: _finish_auth 0 2024-03-25T19:35:30.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.054+0000 7fe6490af640 10 monclient: _check_auth_tickets 2024-03-25T19:35:30.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.054+0000 7fe6490af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:30.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.054+0000 7fe6490af640 1 -- 172.21.15.111:0/639829879 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fe63c001a10 con 0x7fe6440a96b0 2024-03-25T19:35:30.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.054+0000 7fe6490af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:30.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.054+0000 7fe6490af640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:30.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.054+0000 7fe6490af640 10 monclient: dump: 2024-03-25T19:35:30.054 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:30.054 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:30.054 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.054 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.054 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:30.054 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:30.054 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:30.054 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:30.054 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:30.054 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:30.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.054+0000 7fe64c315640 5 monclient: authenticate success, global_id 4133 2024-03-25T19:35:30.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.054+0000 7fe64c315640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:30.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.056+0000 7fe6490af640 1 -- 172.21.15.111:0/639829879 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe63c001b70 con 0x7fe6440a96b0 2024-03-25T19:35:30.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.056+0000 7fe6490af640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:30.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.056+0000 7fe6490af640 1 -- 172.21.15.111:0/639829879 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fe63c001f80 con 0x7fe6440a96b0 2024-03-25T19:35:30.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.056+0000 7fe6490af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:30.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.056+0000 7fe6490af640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:30.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.056+0000 7fe6490af640 10 monclient: dump: 2024-03-25T19:35:30.056 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:30.057 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:30.057 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.057 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.057 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:30.057 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:30.057 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:30.057 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:30.057 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:30.057 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:30.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.056+0000 7fe64c315640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:30.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.056+0000 7fe64c315640 4 set_mon_vals no callback set 2024-03-25T19:35:30.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.056+0000 7fe64c315640 10 monclient: shutdown 2024-03-25T19:35:30.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.056+0000 7fe64c315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:30.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.056+0000 7fe64c315640 1 -- 172.21.15.111:0/639829879 >> v1:172.21.15.44:6790/0 conn(0x7fe6440a96b0 legacy=0x7fe64413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:30.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.056+0000 7fe64c315640 1 -- 172.21.15.111:0/639829879 shutdown_connections 2024-03-25T19:35:30.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.056+0000 7fe64c315640 1 -- 172.21.15.111:0/639829879 >> 172.21.15.111:0/639829879 conn(0x7fe6440668e0 msgr2=0x7fe644066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:30.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.056+0000 7fe64c315640 1 -- 172.21.15.111:0/639829879 shutdown_connections 2024-03-25T19:35:30.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.056+0000 7fe64c315640 1 -- 172.21.15.111:0/639829879 wait complete. 2024-03-25T19:35:30.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.057+0000 7fe64c315640 10 monclient: build_initial_monmap 2024-03-25T19:35:30.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.057+0000 7fe64c315640 10 monclient: monmap: 2024-03-25T19:35:30.058 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T19:35:30.058 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:30.058 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 0.000000 2024-03-25T19:35:30.058 INFO:teuthology.orchestra.run.smithi111.stderr:created 0.000000 2024-03-25T19:35:30.058 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:30.058 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 1 2024-03-25T19:35:30.058 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:30.058 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:35:30.058 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:35:30.058 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:30.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.057+0000 7fe64c315640 1 librados: starting msgr at 2024-03-25T19:35:30.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.057+0000 7fe64c315640 1 librados: starting objecter 2024-03-25T19:35:30.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.057+0000 7fe64c315640 1 Processor -- start 2024-03-25T19:35:30.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.057+0000 7fe64c315640 1 -- start start 2024-03-25T19:35:30.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.057+0000 7fe64c315640 1 librados: setting wanted keys 2024-03-25T19:35:30.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.057+0000 7fe64c315640 1 librados: calling monclient init 2024-03-25T19:35:30.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.057+0000 7fe64c315640 10 monclient: init 2024-03-25T19:35:30.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.058+0000 7fe64c315640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:30.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.058+0000 7fe64c315640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:35:30.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.058+0000 7fe64c315640 10 monclient(hunting): picked mon.noname-a con 0x7fe6440a96b0 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:30.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.058+0000 7fe64c315640 10 monclient(hunting): picked mon.noname-b con 0x7fe6440b4700 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:30.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.058+0000 7fe64c315640 10 monclient(hunting): picked mon.noname-c con 0x7fe644142a20 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:30.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.058+0000 7fe64c315640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe64413a970 con 0x7fe6440a96b0 2024-03-25T19:35:30.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.058+0000 7fe64c315640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6441d47a0 con 0x7fe6440b4700 2024-03-25T19:35:30.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.058+0000 7fe64c315640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe64413b4e0 con 0x7fe644142a20 2024-03-25T19:35:30.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.058+0000 7fe64c315640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:30.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.058+0000 7fe64c315640 10 monclient(hunting): authenticate will time out at 5351.193295s 2024-03-25T19:35:30.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.058+0000 7fe64a8b2640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fe644142a20 0x7fe64413b950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.111:42436/0 (socket says 172.21.15.111:42436) 2024-03-25T19:35:30.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.059+0000 7fe64a8b2640 1 -- 172.21.15.111:0/2921852260 learned_addr learned my addr 172.21.15.111:0/2921852260 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T19:35:30.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.059+0000 7fe63affd640 10 client.?.objecter ms_handle_connect 0x7fe644142a20 2024-03-25T19:35:30.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.059+0000 7fe63affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:30.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.059+0000 7fe63affd640 10 client.?.objecter ms_handle_connect 0x7fe6440a96b0 2024-03-25T19:35:30.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.059+0000 7fe63affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:30.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.059+0000 7fe63affd640 10 client.?.objecter ms_handle_connect 0x7fe6440b4700 2024-03-25T19:35:30.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.059+0000 7fe63affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:30.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.061+0000 7fe63affd640 1 -- 172.21.15.111:0/2921852260 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063355252 0 0) 0x7fe64413b4e0 con 0x7fe644142a20 2024-03-25T19:35:30.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.061+0000 7fe63affd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:30.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.061+0000 7fe63affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:30.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.061+0000 7fe63affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:30.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.061+0000 7fe63affd640 10 monclient(hunting): my global_id is 4136 2024-03-25T19:35:30.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.061+0000 7fe63affd640 1 -- 172.21.15.111:0/2921852260 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe628004eb0 con 0x7fe644142a20 2024-03-25T19:35:30.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.061+0000 7fe63affd640 1 -- 172.21.15.111:0/2921852260 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3315718042 0 0) 0x7fe6441d47a0 con 0x7fe6440b4700 2024-03-25T19:35:30.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.061+0000 7fe63affd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:30.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.061+0000 7fe63affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:30.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.061+0000 7fe63affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:30.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.061+0000 7fe63affd640 10 monclient(hunting): my global_id is 4135 2024-03-25T19:35:30.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.061+0000 7fe63affd640 1 -- 172.21.15.111:0/2921852260 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe64413b4e0 con 0x7fe6440b4700 2024-03-25T19:35:30.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.061+0000 7fe63affd640 1 -- 172.21.15.111:0/2921852260 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3225429351 0 0) 0x7fe6440aa1f0 con 0x7fe6440a96b0 2024-03-25T19:35:30.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.061+0000 7fe63affd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:30.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.061+0000 7fe63affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:30.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.061+0000 7fe63affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:30.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.061+0000 7fe63affd640 10 monclient(hunting): my global_id is 4137 2024-03-25T19:35:30.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.061+0000 7fe63affd640 1 -- 172.21.15.111:0/2921852260 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6441d47a0 con 0x7fe6440a96b0 2024-03-25T19:35:30.065 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.066+0000 7fe63affd640 1 -- 172.21.15.111:0/2921852260 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 575092225 0 0) 0x7fe64413b4e0 con 0x7fe6440b4700 2024-03-25T19:35:30.065 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.066+0000 7fe63affd640 1 -- 172.21.15.111:0/2921852260 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6440aa1f0 con 0x7fe6440b4700 2024-03-25T19:35:30.065 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.066+0000 7fe63affd640 1 -- 172.21.15.111:0/2921852260 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fe634001870 con 0x7fe6440b4700 2024-03-25T19:35:30.065 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.066+0000 7fe63affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:30.065 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.066+0000 7fe63affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:30.066 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.066+0000 7fe63affd640 10 monclient(hunting): dump: 2024-03-25T19:35:30.066 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:30.066 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:30.066 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.066 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.066 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:30.066 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:30.066 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:30.066 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:30.066 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:30.066 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:30.066 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.066+0000 7fe63affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:30.066 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.066+0000 7fe63affd640 1 -- 172.21.15.111:0/2921852260 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3564686048 0 0) 0x7fe6441d47a0 con 0x7fe6440a96b0 2024-03-25T19:35:30.066 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.066+0000 7fe63affd640 1 -- 172.21.15.111:0/2921852260 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe64413b4e0 con 0x7fe6440a96b0 2024-03-25T19:35:30.066 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.066+0000 7fe63affd640 1 -- 172.21.15.111:0/2921852260 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fe640001750 con 0x7fe6440a96b0 2024-03-25T19:35:30.066 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.066+0000 7fe63affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:30.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.066+0000 7fe63affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:30.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.066+0000 7fe63affd640 10 monclient(hunting): dump: 2024-03-25T19:35:30.067 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:30.067 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:30.067 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.067 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.067 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:30.067 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:30.067 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:30.067 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:30.067 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:30.067 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:30.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.067+0000 7fe63affd640 1 -- 172.21.15.111:0/2921852260 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2633171143 0 0) 0x7fe628004eb0 con 0x7fe644142a20 2024-03-25T19:35:30.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.067+0000 7fe63affd640 1 -- 172.21.15.111:0/2921852260 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6441d47a0 con 0x7fe644142a20 2024-03-25T19:35:30.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.067+0000 7fe63affd640 1 -- 172.21.15.111:0/2921852260 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fe63c001ec0 con 0x7fe644142a20 2024-03-25T19:35:30.068 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.067+0000 7fe63affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:30.068 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.067+0000 7fe63affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:30.068 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.067+0000 7fe63affd640 10 monclient(hunting): dump: 2024-03-25T19:35:30.068 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:30.068 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:30.068 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.068 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.068 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:30.068 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:30.068 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:30.068 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:30.068 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:30.068 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:30.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.069+0000 7fe63affd640 1 -- 172.21.15.111:0/2921852260 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2083810609 0 0) 0x7fe64413b4e0 con 0x7fe6440a96b0 2024-03-25T19:35:30.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.069+0000 7fe63affd640 1 -- 172.21.15.111:0/2921852260 >> v1:172.21.15.44:6790/0 conn(0x7fe644142a20 legacy=0x7fe64413b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:30.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.069+0000 7fe63affd640 1 -- 172.21.15.111:0/2921852260 >> v1:172.21.15.111:6789/0 conn(0x7fe6440b4700 legacy=0x7fe64413add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:30.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.069+0000 7fe63affd640 10 monclient: _finish_hunting 0 2024-03-25T19:35:30.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.069+0000 7fe63affd640 1 monclient: found mon.a 2024-03-25T19:35:30.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.069+0000 7fe63affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:30.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.069+0000 7fe63affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:30.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.069+0000 7fe63affd640 1 -- 172.21.15.111:0/2921852260 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6441d8030 con 0x7fe6440a96b0 2024-03-25T19:35:30.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.069+0000 7fe63affd640 10 monclient: _finish_auth 0 2024-03-25T19:35:30.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.069+0000 7fe63affd640 10 monclient: _check_auth_tickets 2024-03-25T19:35:30.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.069+0000 7fe63affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:30.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.069+0000 7fe64c315640 5 monclient: authenticate success, global_id 4137 2024-03-25T19:35:30.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.069+0000 7fe64c315640 10 monclient: _renew_subs 2024-03-25T19:35:30.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.069+0000 7fe64c315640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:30.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.069+0000 7fe64c315640 1 -- 172.21.15.111:0/2921852260 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6441d82a0 con 0x7fe6440a96b0 2024-03-25T19:35:30.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.069+0000 7fe64c315640 10 client.4137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:30.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.069+0000 7fe64c315640 10 monclient: _renew_subs 2024-03-25T19:35:30.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.069+0000 7fe64c315640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:30.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.069+0000 7fe64c315640 1 -- 172.21.15.111:0/2921852260 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe64413c240 con 0x7fe6440a96b0 2024-03-25T19:35:30.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.069+0000 7fe64c315640 1 librados: init done 2024-03-25T19:35:30.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.070+0000 7fe64c315640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:30.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.070+0000 7fe64c315640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:30.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.070+0000 7fe64c315640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:30.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.070+0000 7fe64c315640 1 -- 172.21.15.111:0/2921852260 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe64404f730 con 0x7fe6440a96b0 2024-03-25T19:35:30.072 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T19:35:30.071+0000 7f54ee4c7040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:35:30.103 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:30.104+0000 7f882064b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:35:30.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.175+0000 7fe63affd640 1 -- 172.21.15.111:0/2921852260 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe640001be0 con 0x7fe6440a96b0 2024-03-25T19:35:30.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.175+0000 7fe63affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:30.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.175+0000 7fe63affd640 1 -- 172.21.15.111:0/2921852260 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fe640003790 con 0x7fe6440a96b0 2024-03-25T19:35:30.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.175+0000 7fe63affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:30.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.175+0000 7fe63affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:30.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.175+0000 7fe63affd640 10 monclient: dump: 2024-03-25T19:35:30.174 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:30.174 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:30.174 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.174 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.174 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:30.174 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:30.174 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:30.174 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:30.175 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:30.175 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:30.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.175+0000 7fe63affd640 1 -- 172.21.15.111:0/2921852260 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe640003c80 con 0x7fe6440a96b0 2024-03-25T19:35:30.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.175+0000 7fe6488ae640 4 set_mon_vals no callback set 2024-03-25T19:35:30.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.176+0000 7fe63affd640 1 -- 172.21.15.111:0/2921852260 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3620127884 0 0) 0x7fe6400051e0 con 0x7fe6440a96b0 2024-03-25T19:35:30.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.176+0000 7fe63affd640 10 client.4137.objecter ms_dispatch 0x7fe644000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T19:35:30.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.176+0000 7fe63affd640 3 client.4137.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T19:35:30.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.176+0000 7fe63affd640 3 client.4137.objecter handle_osd_map decoding full epoch 5 2024-03-25T19:35:30.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.176+0000 7fe63affd640 20 client.4137.objecter dump_active .. 0 homeless 2024-03-25T19:35:30.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.176+0000 7fe63affd640 1 -- 172.21.15.111:0/2921852260 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe640004d70 con 0x7fe6440a96b0 2024-03-25T19:35:30.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.176+0000 7fe63affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:30.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.176+0000 7fe63affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:30.189 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T19:35:30.189+0000 7f54ee4c7040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:35:30.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.249+0000 7fe64c315640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c938187c-9648-474d-896c-4e91867dfc72", "id": 3}] 2024-03-25T19:35:30.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.249+0000 7fe64c315640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c938187c-9648-474d-896c-4e91867dfc72", "id": 3}] 2024-03-25T19:35:30.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.249+0000 7fe64c315640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:30.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.249+0000 7fe64c315640 1 -- 172.21.15.111:0/2921852260 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c938187c-9648-474d-896c-4e91867dfc72", "id": 3} v 0) v1 -- 0x7fe644069f30 con 0x7fe6440a96b0 2024-03-25T19:35:30.260 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:30.261+0000 7f882064b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:35:30.293 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T19:35:30.293+0000 7f54ee4c7040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:35:30.301 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.303+0000 7fe63affd640 1 -- 172.21.15.111:0/2921852260 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c938187c-9648-474d-896c-4e91867dfc72", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 2020247036 0 525046306) 0x7fe6400040e0 con 0x7fe6440a96b0 2024-03-25T19:35:30.302 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.303+0000 7fe63affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c938187c-9648-474d-896c-4e91867dfc72", "id": 3}] 2024-03-25T19:35:30.302 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.303+0000 7fe63affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:30.302 INFO:teuthology.orchestra.run.smithi111.stdout:3 2024-03-25T19:35:30.303 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.304+0000 7fe64c315640 10 librados: watch_flush enter 2024-03-25T19:35:30.303 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.304+0000 7fe64c315640 10 librados: watch_flush exit 2024-03-25T19:35:30.303 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.304+0000 7fe64c315640 20 client.4137.objecter shutdown clearing up homeless session... 2024-03-25T19:35:30.304 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.304+0000 7fe64c315640 10 client.4137.objecter successfully canceled tick 2024-03-25T19:35:30.304 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.304+0000 7fe64c315640 10 monclient: shutdown 2024-03-25T19:35:30.304 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.304+0000 7fe64c315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:30.304 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.304+0000 7fe64c315640 1 -- 172.21.15.111:0/2921852260 >> v1:172.21.15.44:6789/0 conn(0x7fe6440a96b0 legacy=0x7fe6441d3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:30.304 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.304+0000 7fe64c315640 1 -- 172.21.15.111:0/2921852260 shutdown_connections 2024-03-25T19:35:30.304 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.304+0000 7fe64c315640 1 -- 172.21.15.111:0/2921852260 >> 172.21.15.111:0/2921852260 conn(0x7fe6440668e0 msgr2=0x7fe6440b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:30.304 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.304+0000 7fe64c315640 1 -- 172.21.15.111:0/2921852260 shutdown_connections 2024-03-25T19:35:30.304 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.304+0000 7fe64c315640 1 -- 172.21.15.111:0/2921852260 wait complete. 2024-03-25T19:35:30.304 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.304+0000 7fe64c315640 1 librados: shutdown 2024-03-25T19:35:30.304 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:30.306+0000 7f882064b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:35:30.313 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new 3616d04d-8397-4d67-afb7-b5f914f4a7bd 4 2024-03-25T19:35:30.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.412+0000 7fcd9bc81640 10 monclient: get_monmap_and_config 2024-03-25T19:35:30.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.412+0000 7fcd9bc81640 10 monclient: build_initial_monmap 2024-03-25T19:35:30.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.412+0000 7fcd9bc81640 10 monclient: monmap: 2024-03-25T19:35:30.412 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T19:35:30.412 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:30.412 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:35:30.413439+0000 2024-03-25T19:35:30.412 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:35:30.413439+0000 2024-03-25T19:35:30.412 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:30.412 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:30.413 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:30.413 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:35:30.413 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:35:30.413 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:30.414 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.412+0000 7fcd9bc81640 1 Processor -- start 2024-03-25T19:35:30.414 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.413+0000 7fcd9bc81640 1 -- start start 2024-03-25T19:35:30.414 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.413+0000 7fcd9bc81640 10 monclient: init 2024-03-25T19:35:30.414 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.413+0000 7fcd9bc81640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:30.414 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.413+0000 7fcd9bc81640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:35:30.415 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.413+0000 7fcd9bc81640 10 monclient(hunting): picked mon.noname-b con 0x7fcd940b4700 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:30.415 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.413+0000 7fcd9bc81640 10 monclient(hunting): picked mon.noname-a con 0x7fcd94142a20 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:30.415 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.413+0000 7fcd9bc81640 10 monclient(hunting): picked mon.noname-c con 0x7fcd940a96b0 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:30.415 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.413+0000 7fcd9bc81640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd940a9d60 con 0x7fcd94142a20 2024-03-25T19:35:30.415 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.413+0000 7fcd9bc81640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd9413ff10 con 0x7fcd940a96b0 2024-03-25T19:35:30.415 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.413+0000 7fcd9bc81640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd940aa1f0 con 0x7fcd940b4700 2024-03-25T19:35:30.415 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.413+0000 7fcd9bc81640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:30.415 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.413+0000 7fcd9bc81640 10 monclient(hunting): authenticate will time out at 5351.549566s 2024-03-25T19:35:30.415 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.414+0000 7fcd99a1d640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fcd940b4700 0x7fcd94142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.111:42450/0 (socket says 172.21.15.111:42450) 2024-03-25T19:35:30.415 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.414+0000 7fcd99a1d640 1 -- 172.21.15.111:0/429295272 learned_addr learned my addr 172.21.15.111:0/429295272 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T19:35:30.417 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.416+0000 7fcd98a1b640 1 -- 172.21.15.111:0/429295272 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104578448 0 0) 0x7fcd940a9d60 con 0x7fcd94142a20 2024-03-25T19:35:30.417 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.416+0000 7fcd98a1b640 10 monclient(hunting): _negotiate 2024-03-25T19:35:30.417 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.416+0000 7fcd98a1b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:30.417 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.416+0000 7fcd98a1b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:30.417 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.416+0000 7fcd98a1b640 10 monclient(hunting): my global_id is 4140 2024-03-25T19:35:30.417 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.416+0000 7fcd98a1b640 1 -- 172.21.15.111:0/429295272 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd80003520 con 0x7fcd94142a20 2024-03-25T19:35:30.417 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.417+0000 7fcd98a1b640 1 -- 172.21.15.111:0/429295272 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206576149 0 0) 0x7fcd940aa1f0 con 0x7fcd940b4700 2024-03-25T19:35:30.417 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.417+0000 7fcd98a1b640 10 monclient(hunting): _negotiate 2024-03-25T19:35:30.417 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.417+0000 7fcd98a1b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:30.417 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.417+0000 7fcd98a1b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:30.417 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.417+0000 7fcd98a1b640 10 monclient(hunting): my global_id is 4139 2024-03-25T19:35:30.417 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.417+0000 7fcd98a1b640 1 -- 172.21.15.111:0/429295272 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd940a9d60 con 0x7fcd940b4700 2024-03-25T19:35:30.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.417+0000 7fcd98a1b640 1 -- 172.21.15.111:0/429295272 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4007081278 0 0) 0x7fcd9413ff10 con 0x7fcd940a96b0 2024-03-25T19:35:30.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.417+0000 7fcd98a1b640 10 monclient(hunting): _negotiate 2024-03-25T19:35:30.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.417+0000 7fcd98a1b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:30.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.417+0000 7fcd98a1b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:30.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.417+0000 7fcd98a1b640 10 monclient(hunting): my global_id is 4138 2024-03-25T19:35:30.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.417+0000 7fcd98a1b640 1 -- 172.21.15.111:0/429295272 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd940aa1f0 con 0x7fcd940a96b0 2024-03-25T19:35:30.420 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.420+0000 7fcd98a1b640 1 -- 172.21.15.111:0/429295272 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 75190798 0 0) 0x7fcd80003520 con 0x7fcd94142a20 2024-03-25T19:35:30.420 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.420+0000 7fcd98a1b640 1 -- 172.21.15.111:0/429295272 >> v1:172.21.15.44:6790/0 conn(0x7fcd940b4700 legacy=0x7fcd94142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:30.420 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.420+0000 7fcd98a1b640 1 -- 172.21.15.111:0/429295272 >> v1:172.21.15.111:6789/0 conn(0x7fcd940a96b0 legacy=0x7fcd9413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:30.420 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.420+0000 7fcd98a1b640 10 monclient: _finish_hunting 0 2024-03-25T19:35:30.420 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.420+0000 7fcd98a1b640 1 monclient: found mon.noname-a 2024-03-25T19:35:30.420 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.420+0000 7fcd98a1b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:30.420 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.420+0000 7fcd98a1b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:35:30.420 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.420+0000 7fcd98a1b640 1 -- 172.21.15.111:0/429295272 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd940b4ae0 con 0x7fcd94142a20 2024-03-25T19:35:30.420 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.420+0000 7fcd98a1b640 10 monclient: _finish_auth 0 2024-03-25T19:35:30.421 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.420+0000 7fcd98a1b640 10 monclient: _check_auth_tickets 2024-03-25T19:35:30.421 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.420+0000 7fcd98a1b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:30.421 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.420+0000 7fcd98a1b640 1 -- 172.21.15.111:0/429295272 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fcd84001690 con 0x7fcd94142a20 2024-03-25T19:35:30.421 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.420+0000 7fcd9bc81640 5 monclient: authenticate success, global_id 4140 2024-03-25T19:35:30.421 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.420+0000 7fcd9bc81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:30.421 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.420+0000 7fcd98a1b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:30.421 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.420+0000 7fcd98a1b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:30.421 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.420+0000 7fcd98a1b640 10 monclient: dump: 2024-03-25T19:35:30.421 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:30.421 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:30.421 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.421 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.421 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:30.422 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:30.422 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:30.422 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:30.422 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:30.422 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:30.422 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.420+0000 7fcd9bc81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:30.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.421+0000 7fcd98a1b640 1 -- 172.21.15.111:0/429295272 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd840017f0 con 0x7fcd94142a20 2024-03-25T19:35:30.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.421+0000 7fcd98a1b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:30.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.421+0000 7fcd98a1b640 1 -- 172.21.15.111:0/429295272 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fcd84001c00 con 0x7fcd94142a20 2024-03-25T19:35:30.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.421+0000 7fcd9bc81640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:30.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.421+0000 7fcd9bc81640 4 set_mon_vals no callback set 2024-03-25T19:35:30.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.421+0000 7fcd98a1b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:30.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.421+0000 7fcd98a1b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:30.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.421+0000 7fcd98a1b640 10 monclient: dump: 2024-03-25T19:35:30.425 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:30.426 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:30.426 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.426 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.426 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:30.426 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:30.426 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:30.426 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:30.426 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:30.426 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:30.426 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.422+0000 7fcd9bc81640 10 monclient: shutdown 2024-03-25T19:35:30.426 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.422+0000 7fcd9bc81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:30.426 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.422+0000 7fcd9bc81640 1 -- 172.21.15.111:0/429295272 >> v1:172.21.15.44:6789/0 conn(0x7fcd94142a20 legacy=0x7fcd9413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:30.426 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.422+0000 7fcd9bc81640 1 -- 172.21.15.111:0/429295272 shutdown_connections 2024-03-25T19:35:30.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.422+0000 7fcd9bc81640 1 -- 172.21.15.111:0/429295272 >> 172.21.15.111:0/429295272 conn(0x7fcd940668e0 msgr2=0x7fcd94066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:30.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.422+0000 7fcd9bc81640 1 -- 172.21.15.111:0/429295272 shutdown_connections 2024-03-25T19:35:30.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.422+0000 7fcd9bc81640 1 -- 172.21.15.111:0/429295272 wait complete. 2024-03-25T19:35:30.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.423+0000 7fcd9bc81640 10 monclient: build_initial_monmap 2024-03-25T19:35:30.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.423+0000 7fcd9bc81640 10 monclient: monmap: 2024-03-25T19:35:30.427 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T19:35:30.427 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:30.427 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 0.000000 2024-03-25T19:35:30.427 INFO:teuthology.orchestra.run.smithi111.stderr:created 0.000000 2024-03-25T19:35:30.427 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:30.427 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 1 2024-03-25T19:35:30.427 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:30.427 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:35:30.427 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:35:30.427 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:30.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.423+0000 7fcd9bc81640 1 librados: starting msgr at 2024-03-25T19:35:30.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.423+0000 7fcd9bc81640 1 librados: starting objecter 2024-03-25T19:35:30.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.423+0000 7fcd9bc81640 1 Processor -- start 2024-03-25T19:35:30.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.423+0000 7fcd9bc81640 1 -- start start 2024-03-25T19:35:30.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.423+0000 7fcd9bc81640 1 librados: setting wanted keys 2024-03-25T19:35:30.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.423+0000 7fcd9bc81640 1 librados: calling monclient init 2024-03-25T19:35:30.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.423+0000 7fcd9bc81640 10 monclient: init 2024-03-25T19:35:30.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.424+0000 7fcd9bc81640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:30.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.424+0000 7fcd9bc81640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:30.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.424+0000 7fcd9bc81640 10 monclient(hunting): picked mon.noname-b con 0x7fcd940a96b0 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:30.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.424+0000 7fcd9bc81640 10 monclient(hunting): picked mon.noname-a con 0x7fcd940b4700 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:30.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.424+0000 7fcd9bc81640 10 monclient(hunting): picked mon.noname-c con 0x7fcd94142a20 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:30.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.424+0000 7fcd9bc81640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd9413b5d0 con 0x7fcd940b4700 2024-03-25T19:35:30.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.424+0000 7fcd9bc81640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd9413c480 con 0x7fcd940a96b0 2024-03-25T19:35:30.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.424+0000 7fcd9bc81640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd9413b850 con 0x7fcd94142a20 2024-03-25T19:35:30.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.424+0000 7fcd9bc81640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:30.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.424+0000 7fcd9bc81640 10 monclient(hunting): authenticate will time out at 5351.560007s 2024-03-25T19:35:30.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.424+0000 7fcd9921c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fcd940b4700 0x7fcd941dabd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.111:44304/0 (socket says 172.21.15.111:44304) 2024-03-25T19:35:30.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.424+0000 7fcd9921c640 1 -- 172.21.15.111:0/1584606542 learned_addr learned my addr 172.21.15.111:0/1584606542 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T19:35:30.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.424+0000 7fcd8a7fc640 10 client.?.objecter ms_handle_connect 0x7fcd94142a20 2024-03-25T19:35:30.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.425+0000 7fcd8a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:30.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.425+0000 7fcd8a7fc640 10 client.?.objecter ms_handle_connect 0x7fcd940b4700 2024-03-25T19:35:30.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.425+0000 7fcd8a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:30.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.427+0000 7fcd8a7fc640 1 -- 172.21.15.111:0/1584606542 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3138332865 0 0) 0x7fcd9413b5d0 con 0x7fcd940b4700 2024-03-25T19:35:30.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.427+0000 7fcd8a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:30.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.427+0000 7fcd8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:30.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.427+0000 7fcd8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:30.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.427+0000 7fcd8a7fc640 10 monclient(hunting): my global_id is 4143 2024-03-25T19:35:30.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.427+0000 7fcd8a7fc640 1 -- 172.21.15.111:0/1584606542 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd78004eb0 con 0x7fcd940b4700 2024-03-25T19:35:30.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.427+0000 7fcd8a7fc640 1 -- 172.21.15.111:0/1584606542 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3788833728 0 0) 0x7fcd9413b850 con 0x7fcd94142a20 2024-03-25T19:35:30.431 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.427+0000 7fcd8a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:30.431 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.427+0000 7fcd8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:30.431 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.427+0000 7fcd8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:30.431 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.427+0000 7fcd8a7fc640 10 monclient(hunting): my global_id is 4142 2024-03-25T19:35:30.431 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.427+0000 7fcd8a7fc640 1 -- 172.21.15.111:0/1584606542 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd9413b5d0 con 0x7fcd94142a20 2024-03-25T19:35:30.431 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.429+0000 7fcd8a7fc640 10 client.?.objecter ms_handle_connect 0x7fcd940a96b0 2024-03-25T19:35:30.431 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.429+0000 7fcd8a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:30.432 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.431+0000 7fcd8a7fc640 1 -- 172.21.15.111:0/1584606542 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115551252 0 0) 0x7fcd9413c480 con 0x7fcd940a96b0 2024-03-25T19:35:30.432 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.431+0000 7fcd8a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:30.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.431+0000 7fcd8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:30.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.431+0000 7fcd8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:30.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.431+0000 7fcd8a7fc640 10 monclient(hunting): my global_id is 4141 2024-03-25T19:35:30.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.431+0000 7fcd8a7fc640 1 -- 172.21.15.111:0/1584606542 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd9413b850 con 0x7fcd940a96b0 2024-03-25T19:35:30.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.431+0000 7fcd8a7fc640 1 -- 172.21.15.111:0/1584606542 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3899949538 0 0) 0x7fcd78004eb0 con 0x7fcd940b4700 2024-03-25T19:35:30.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.431+0000 7fcd8a7fc640 1 -- 172.21.15.111:0/1584606542 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd9413c480 con 0x7fcd940b4700 2024-03-25T19:35:30.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.431+0000 7fcd8a7fc640 1 -- 172.21.15.111:0/1584606542 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fcd84001bc0 con 0x7fcd940b4700 2024-03-25T19:35:30.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.431+0000 7fcd8a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:30.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.431+0000 7fcd8a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:30.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.431+0000 7fcd8a7fc640 10 monclient(hunting): dump: 2024-03-25T19:35:30.433 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:30.433 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:30.433 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.433 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.433 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:30.434 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:30.434 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:30.434 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:30.434 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:30.434 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:30.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.431+0000 7fcd8a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:30.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.432+0000 7fcd8a7fc640 1 -- 172.21.15.111:0/1584606542 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 693041757 0 0) 0x7fcd9413b5d0 con 0x7fcd94142a20 2024-03-25T19:35:30.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.432+0000 7fcd8a7fc640 1 -- 172.21.15.111:0/1584606542 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd78004eb0 con 0x7fcd94142a20 2024-03-25T19:35:30.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.433+0000 7fcd8a7fc640 1 -- 172.21.15.111:0/1584606542 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fcd8c001bd0 con 0x7fcd94142a20 2024-03-25T19:35:30.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.433+0000 7fcd8a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:30.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.433+0000 7fcd8a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:30.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.433+0000 7fcd8a7fc640 10 monclient(hunting): dump: 2024-03-25T19:35:30.434 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:30.434 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:30.435 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.435 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.435 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:30.435 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:30.435 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:30.435 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:30.435 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:30.435 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:30.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.435+0000 7fcd8a7fc640 1 -- 172.21.15.111:0/1584606542 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2451233530 0 0) 0x7fcd9413c480 con 0x7fcd940b4700 2024-03-25T19:35:30.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.435+0000 7fcd8a7fc640 1 -- 172.21.15.111:0/1584606542 >> v1:172.21.15.44:6790/0 conn(0x7fcd94142a20 legacy=0x7fcd941db2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.435+0000 7fcd8a7fc640 1 -- 172.21.15.111:0/1584606542 >> v1:172.21.15.111:6789/0 conn(0x7fcd940a96b0 legacy=0x7fcd9413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.435+0000 7fcd8a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:35:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.435+0000 7fcd8a7fc640 1 monclient: found mon.a 2024-03-25T19:35:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.435+0000 7fcd8a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.435+0000 7fcd8a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.435+0000 7fcd8a7fc640 1 -- 172.21.15.111:0/1584606542 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd941d4fe0 con 0x7fcd940b4700 2024-03-25T19:35:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.435+0000 7fcd8a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:35:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.435+0000 7fcd8a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:35:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.435+0000 7fcd8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.435+0000 7fcd9bc81640 5 monclient: authenticate success, global_id 4143 2024-03-25T19:35:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.435+0000 7fcd9bc81640 10 monclient: _renew_subs 2024-03-25T19:35:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.435+0000 7fcd9bc81640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.435+0000 7fcd9bc81640 1 -- 172.21.15.111:0/1584606542 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd941e2ca0 con 0x7fcd940b4700 2024-03-25T19:35:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.435+0000 7fcd9bc81640 10 client.4143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:30.438 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.435+0000 7fcd9bc81640 10 monclient: _renew_subs 2024-03-25T19:35:30.438 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.435+0000 7fcd9bc81640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:30.438 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.435+0000 7fcd9bc81640 1 -- 172.21.15.111:0/1584606542 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd941e3010 con 0x7fcd940b4700 2024-03-25T19:35:30.438 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.435+0000 7fcd9bc81640 1 librados: init done 2024-03-25T19:35:30.438 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.436+0000 7fcd9bc81640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:30.438 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.436+0000 7fcd9bc81640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:30.438 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.436+0000 7fcd9bc81640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:30.438 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.436+0000 7fcd9bc81640 1 -- 172.21.15.111:0/1584606542 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd9404f730 con 0x7fcd940b4700 2024-03-25T19:35:30.461 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T19:35:30.461+0000 7f54ee4c7040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:35:30.473 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:30.473+0000 7f882064b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:35:30.505 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T19:35:30.505+0000 7f54ee4c7040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:35:30.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.536+0000 7fcd8a7fc640 1 -- 172.21.15.111:0/1584606542 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd840011f0 con 0x7fcd940b4700 2024-03-25T19:35:30.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.536+0000 7fcd8a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:30.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.536+0000 7fcd8a7fc640 1 -- 172.21.15.111:0/1584606542 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fcd84003b00 con 0x7fcd940b4700 2024-03-25T19:35:30.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.536+0000 7fcd8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:30.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.536+0000 7fcd8a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:30.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.536+0000 7fcd8a7fc640 10 monclient: dump: 2024-03-25T19:35:30.536 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:30.536 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:30.536 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.536 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.536 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:30.537 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:30.537 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:30.537 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:30.537 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:30.537 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:30.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.536+0000 7fcd8a7fc640 1 -- 172.21.15.111:0/1584606542 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fcd840040b0 con 0x7fcd940b4700 2024-03-25T19:35:30.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.536+0000 7fcd8bfff640 4 set_mon_vals no callback set 2024-03-25T19:35:30.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.537+0000 7fcd8a7fc640 1 -- 172.21.15.111:0/1584606542 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 1561489731 0 0) 0x7fcd84006820 con 0x7fcd940b4700 2024-03-25T19:35:30.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.537+0000 7fcd8a7fc640 10 client.4143.objecter ms_dispatch 0x7fcd94000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T19:35:30.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.537+0000 7fcd8a7fc640 3 client.4143.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T19:35:30.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.537+0000 7fcd8a7fc640 3 client.4143.objecter handle_osd_map decoding full epoch 6 2024-03-25T19:35:30.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.537+0000 7fcd8a7fc640 20 client.4143.objecter dump_active .. 0 homeless 2024-03-25T19:35:30.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.537+0000 7fcd8a7fc640 1 -- 172.21.15.111:0/1584606542 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fcd84006b10 con 0x7fcd940b4700 2024-03-25T19:35:30.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.537+0000 7fcd8a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:30.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.537+0000 7fcd8a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:30.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.609+0000 7fcd9bc81640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3616d04d-8397-4d67-afb7-b5f914f4a7bd", "id": 4}] 2024-03-25T19:35:30.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.609+0000 7fcd9bc81640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3616d04d-8397-4d67-afb7-b5f914f4a7bd", "id": 4}] 2024-03-25T19:35:30.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.609+0000 7fcd9bc81640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:30.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.609+0000 7fcd9bc81640 1 -- 172.21.15.111:0/1584606542 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd new", "uuid": "3616d04d-8397-4d67-afb7-b5f914f4a7bd", "id": 4} v 0) v1 -- 0x7fcd94069f30 con 0x7fcd940b4700 2024-03-25T19:35:30.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.636+0000 7fcd8a7fc640 1 -- 172.21.15.111:0/1584606542 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3616d04d-8397-4d67-afb7-b5f914f4a7bd", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 2634026764 0 1697012583) 0x7fcd840063e0 con 0x7fcd940b4700 2024-03-25T19:35:30.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.636+0000 7fcd8a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3616d04d-8397-4d67-afb7-b5f914f4a7bd", "id": 4}] 2024-03-25T19:35:30.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.636+0000 7fcd8a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:30.636 INFO:teuthology.orchestra.run.smithi111.stdout:4 2024-03-25T19:35:30.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.637+0000 7fcd9bc81640 10 librados: watch_flush enter 2024-03-25T19:35:30.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.637+0000 7fcd9bc81640 10 librados: watch_flush exit 2024-03-25T19:35:30.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.637+0000 7fcd9bc81640 20 client.4143.objecter shutdown clearing up homeless session... 2024-03-25T19:35:30.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.637+0000 7fcd9bc81640 10 client.4143.objecter successfully canceled tick 2024-03-25T19:35:30.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.637+0000 7fcd9bc81640 10 monclient: shutdown 2024-03-25T19:35:30.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.637+0000 7fcd9bc81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:30.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.637+0000 7fcd9bc81640 1 -- 172.21.15.111:0/1584606542 >> v1:172.21.15.44:6789/0 conn(0x7fcd940b4700 legacy=0x7fcd941dabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:30.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.637+0000 7fcd9bc81640 1 -- 172.21.15.111:0/1584606542 shutdown_connections 2024-03-25T19:35:30.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.637+0000 7fcd9bc81640 1 -- 172.21.15.111:0/1584606542 >> 172.21.15.111:0/1584606542 conn(0x7fcd940668e0 msgr2=0x7fcd940b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:30.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.637+0000 7fcd9bc81640 1 -- 172.21.15.111:0/1584606542 shutdown_connections 2024-03-25T19:35:30.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.637+0000 7fcd9bc81640 1 -- 172.21.15.111:0/1584606542 wait complete. 2024-03-25T19:35:30.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.637+0000 7fcd9bc81640 1 librados: shutdown 2024-03-25T19:35:30.647 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new ecd09f68-7386-4523-bf9f-f14877e2e746 5 2024-03-25T19:35:30.668 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T19:35:30.669+0000 7f54ee4c7040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:35:30.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.747+0000 7ffb1dad3640 10 monclient: get_monmap_and_config 2024-03-25T19:35:30.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.747+0000 7ffb1dad3640 10 monclient: build_initial_monmap 2024-03-25T19:35:30.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.747+0000 7ffb1dad3640 10 monclient: monmap: 2024-03-25T19:35:30.747 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T19:35:30.747 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:30.747 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:35:30.748643+0000 2024-03-25T19:35:30.747 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:35:30.748643+0000 2024-03-25T19:35:30.747 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:30.747 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:30.747 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:30.747 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:35:30.747 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:35:30.747 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:30.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.748+0000 7ffb1dad3640 1 Processor -- start 2024-03-25T19:35:30.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.748+0000 7ffb1dad3640 1 -- start start 2024-03-25T19:35:30.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.748+0000 7ffb1dad3640 10 monclient: init 2024-03-25T19:35:30.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.748+0000 7ffb1dad3640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:30.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.748+0000 7ffb1dad3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:35:30.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.748+0000 7ffb1dad3640 10 monclient(hunting): picked mon.noname-b con 0x7ffb180b4700 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:30.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.748+0000 7ffb1dad3640 10 monclient(hunting): picked mon.noname-a con 0x7ffb18142a20 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:30.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.748+0000 7ffb1dad3640 10 monclient(hunting): picked mon.noname-c con 0x7ffb180a96b0 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:30.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.748+0000 7ffb1dad3640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb180a9d60 con 0x7ffb18142a20 2024-03-25T19:35:30.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.748+0000 7ffb1dad3640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1813ff10 con 0x7ffb180a96b0 2024-03-25T19:35:30.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.748+0000 7ffb1dad3640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb180aa1f0 con 0x7ffb180b4700 2024-03-25T19:35:30.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.748+0000 7ffb1dad3640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:30.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.748+0000 7ffb1dad3640 10 monclient(hunting): authenticate will time out at 5351.884723s 2024-03-25T19:35:30.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.749+0000 7ffb177fe640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7ffb180b4700 0x7ffb18142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.111:42464/0 (socket says 172.21.15.111:42464) 2024-03-25T19:35:30.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.749+0000 7ffb177fe640 1 -- 172.21.15.111:0/203285170 learned_addr learned my addr 172.21.15.111:0/203285170 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T19:35:30.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.753+0000 7ffb167fc640 1 -- 172.21.15.111:0/203285170 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948718300 0 0) 0x7ffb180aa1f0 con 0x7ffb180b4700 2024-03-25T19:35:30.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.753+0000 7ffb167fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:30.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.753+0000 7ffb167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:30.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.753+0000 7ffb167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:30.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.753+0000 7ffb167fc640 10 monclient(hunting): my global_id is 4145 2024-03-25T19:35:30.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.753+0000 7ffb167fc640 1 -- 172.21.15.111:0/203285170 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb00003520 con 0x7ffb180b4700 2024-03-25T19:35:30.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.753+0000 7ffb167fc640 1 -- 172.21.15.111:0/203285170 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579854925 0 0) 0x7ffb180a9d60 con 0x7ffb18142a20 2024-03-25T19:35:30.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.753+0000 7ffb167fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:30.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.753+0000 7ffb167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:30.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.753+0000 7ffb167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:30.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.753+0000 7ffb167fc640 10 monclient(hunting): my global_id is 4146 2024-03-25T19:35:30.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.753+0000 7ffb167fc640 1 -- 172.21.15.111:0/203285170 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb180aa1f0 con 0x7ffb18142a20 2024-03-25T19:35:30.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.758+0000 7ffb167fc640 1 -- 172.21.15.111:0/203285170 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4199973409 0 0) 0x7ffb180aa1f0 con 0x7ffb18142a20 2024-03-25T19:35:30.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.758+0000 7ffb167fc640 1 -- 172.21.15.111:0/203285170 >> v1:172.21.15.44:6790/0 conn(0x7ffb180b4700 legacy=0x7ffb18142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:30.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.758+0000 7ffb167fc640 1 -- 172.21.15.111:0/203285170 >> v1:172.21.15.111:6789/0 conn(0x7ffb180a96b0 legacy=0x7ffb1813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:30.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.758+0000 7ffb167fc640 10 monclient: _finish_hunting 0 2024-03-25T19:35:30.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.758+0000 7ffb167fc640 1 monclient: found mon.noname-a 2024-03-25T19:35:30.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.758+0000 7ffb167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:30.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.758+0000 7ffb167fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:35:30.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.758+0000 7ffb167fc640 1 -- 172.21.15.111:0/203285170 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb180b4ae0 con 0x7ffb18142a20 2024-03-25T19:35:30.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.758+0000 7ffb167fc640 10 monclient: _finish_auth 0 2024-03-25T19:35:30.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.758+0000 7ffb167fc640 10 monclient: _check_auth_tickets 2024-03-25T19:35:30.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.758+0000 7ffb167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:30.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.758+0000 7ffb167fc640 1 -- 172.21.15.111:0/203285170 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ffb08001670 con 0x7ffb18142a20 2024-03-25T19:35:30.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.758+0000 7ffb167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:30.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.758+0000 7ffb167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:30.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.758+0000 7ffb167fc640 10 monclient: dump: 2024-03-25T19:35:30.759 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:30.759 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:30.759 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.759 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.759 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:30.759 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:30.760 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:30.760 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:30.760 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:30.760 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:30.760 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.758+0000 7ffb1dad3640 5 monclient: authenticate success, global_id 4146 2024-03-25T19:35:30.760 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.758+0000 7ffb1dad3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:30.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.762+0000 7ffb167fc640 1 -- 172.21.15.111:0/203285170 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb080017d0 con 0x7ffb18142a20 2024-03-25T19:35:30.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.762+0000 7ffb167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:30.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.762+0000 7ffb167fc640 1 -- 172.21.15.111:0/203285170 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ffb08001ba0 con 0x7ffb18142a20 2024-03-25T19:35:30.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.762+0000 7ffb167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:30.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.762+0000 7ffb167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:30.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.762+0000 7ffb167fc640 10 monclient: dump: 2024-03-25T19:35:30.762 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:30.762 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:30.762 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.762 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.762 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:30.762 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:30.762 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:30.762 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:30.762 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:30.762 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:30.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.762+0000 7ffb1dad3640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:30.763 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.762+0000 7ffb1dad3640 4 set_mon_vals no callback set 2024-03-25T19:35:30.765 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.762+0000 7ffb1dad3640 10 monclient: shutdown 2024-03-25T19:35:30.765 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.762+0000 7ffb1dad3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:30.765 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.762+0000 7ffb1dad3640 1 -- 172.21.15.111:0/203285170 >> v1:172.21.15.44:6789/0 conn(0x7ffb18142a20 legacy=0x7ffb1813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:30.765 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.762+0000 7ffb1dad3640 1 -- 172.21.15.111:0/203285170 shutdown_connections 2024-03-25T19:35:30.765 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.762+0000 7ffb1dad3640 1 -- 172.21.15.111:0/203285170 >> 172.21.15.111:0/203285170 conn(0x7ffb180668e0 msgr2=0x7ffb18066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:30.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.762+0000 7ffb1dad3640 1 -- 172.21.15.111:0/203285170 shutdown_connections 2024-03-25T19:35:30.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.762+0000 7ffb1dad3640 1 -- 172.21.15.111:0/203285170 wait complete. 2024-03-25T19:35:30.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.763+0000 7ffb1dad3640 10 monclient: build_initial_monmap 2024-03-25T19:35:30.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.763+0000 7ffb1dad3640 10 monclient: monmap: 2024-03-25T19:35:30.766 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T19:35:30.766 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:30.766 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 0.000000 2024-03-25T19:35:30.766 INFO:teuthology.orchestra.run.smithi111.stderr:created 0.000000 2024-03-25T19:35:30.766 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:30.766 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 1 2024-03-25T19:35:30.766 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:30.766 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:35:30.766 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:35:30.766 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:30.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.763+0000 7ffb1dad3640 1 librados: starting msgr at 2024-03-25T19:35:30.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.763+0000 7ffb1dad3640 1 librados: starting objecter 2024-03-25T19:35:30.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.763+0000 7ffb1dad3640 1 Processor -- start 2024-03-25T19:35:30.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.764+0000 7ffb1dad3640 1 -- start start 2024-03-25T19:35:30.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.764+0000 7ffb1dad3640 1 librados: setting wanted keys 2024-03-25T19:35:30.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.764+0000 7ffb1dad3640 1 librados: calling monclient init 2024-03-25T19:35:30.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.764+0000 7ffb1dad3640 10 monclient: init 2024-03-25T19:35:30.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.764+0000 7ffb1dad3640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:30.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.764+0000 7ffb1dad3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:30.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.764+0000 7ffb1dad3640 10 monclient(hunting): picked mon.noname-c con 0x7ffb180a96b0 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:30.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.764+0000 7ffb1dad3640 10 monclient(hunting): picked mon.noname-b con 0x7ffb180b4700 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:30.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.764+0000 7ffb1dad3640 10 monclient(hunting): picked mon.noname-a con 0x7ffb18142a20 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:30.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.764+0000 7ffb1dad3640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb180ac320 con 0x7ffb18142a20 2024-03-25T19:35:30.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.764+0000 7ffb1dad3640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb180adea0 con 0x7ffb180b4700 2024-03-25T19:35:30.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.764+0000 7ffb1dad3640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb180ae180 con 0x7ffb180a96b0 2024-03-25T19:35:30.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.764+0000 7ffb1dad3640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:30.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.764+0000 7ffb1dad3640 10 monclient(hunting): authenticate will time out at 5351.900355s 2024-03-25T19:35:30.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.765+0000 7ffb177fe640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7ffb180a96b0 0x7ffb1813aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.111:42466/0 (socket says 172.21.15.111:42466) 2024-03-25T19:35:30.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.765+0000 7ffb177fe640 1 -- 172.21.15.111:0/1855665229 learned_addr learned my addr 172.21.15.111:0/1855665229 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T19:35:30.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.765+0000 7ffafffff640 10 client.?.objecter ms_handle_connect 0x7ffb180a96b0 2024-03-25T19:35:30.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.765+0000 7ffafffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:30.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.765+0000 7ffafffff640 10 client.?.objecter ms_handle_connect 0x7ffb180b4700 2024-03-25T19:35:30.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.765+0000 7ffafffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:30.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.774+0000 7ffafffff640 10 client.?.objecter ms_handle_connect 0x7ffb18142a20 2024-03-25T19:35:30.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.774+0000 7ffafffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:30.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.798+0000 7ffafffff640 1 -- 172.21.15.111:0/1855665229 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963911596 0 0) 0x7ffb180ac320 con 0x7ffb18142a20 2024-03-25T19:35:30.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.798+0000 7ffafffff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:30.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.798+0000 7ffafffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:30.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.798+0000 7ffafffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:30.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.798+0000 7ffafffff640 10 monclient(hunting): my global_id is 4149 2024-03-25T19:35:30.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.798+0000 7ffafffff640 1 -- 172.21.15.111:0/1855665229 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb04004eb0 con 0x7ffb18142a20 2024-03-25T19:35:30.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.798+0000 7ffafffff640 1 -- 172.21.15.111:0/1855665229 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307434555 0 0) 0x7ffb180adea0 con 0x7ffb180b4700 2024-03-25T19:35:30.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.798+0000 7ffafffff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:30.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.798+0000 7ffafffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:30.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.798+0000 7ffafffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:30.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.798+0000 7ffafffff640 10 monclient(hunting): my global_id is 4144 2024-03-25T19:35:30.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.798+0000 7ffafffff640 1 -- 172.21.15.111:0/1855665229 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb180ac320 con 0x7ffb180b4700 2024-03-25T19:35:30.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.800+0000 7ffafffff640 1 -- 172.21.15.111:0/1855665229 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500005782 0 0) 0x7ffb180ae180 con 0x7ffb180a96b0 2024-03-25T19:35:30.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.800+0000 7ffafffff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:30.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.800+0000 7ffafffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:30.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.800+0000 7ffafffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:30.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.800+0000 7ffafffff640 10 monclient(hunting): my global_id is 4148 2024-03-25T19:35:30.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.800+0000 7ffafffff640 1 -- 172.21.15.111:0/1855665229 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb180adea0 con 0x7ffb180a96b0 2024-03-25T19:35:30.802 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.802+0000 7ffafffff640 1 -- 172.21.15.111:0/1855665229 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3928989603 0 0) 0x7ffb180ac320 con 0x7ffb180b4700 2024-03-25T19:35:30.802 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.802+0000 7ffafffff640 1 -- 172.21.15.111:0/1855665229 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb180ae180 con 0x7ffb180b4700 2024-03-25T19:35:30.802 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.802+0000 7ffafffff640 1 -- 172.21.15.111:0/1855665229 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ffb08001ae0 con 0x7ffb180b4700 2024-03-25T19:35:30.802 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.802+0000 7ffafffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:30.802 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.802+0000 7ffafffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:30.802 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.802+0000 7ffafffff640 10 monclient(hunting): dump: 2024-03-25T19:35:30.802 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:30.802 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:30.802 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.802 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.802 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:30.802 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:30.802 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:30.803 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:30.803 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:30.803 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:30.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.802+0000 7ffafffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:30.804 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.804+0000 7ffafffff640 1 -- 172.21.15.111:0/1855665229 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3377590607 0 0) 0x7ffb04004eb0 con 0x7ffb18142a20 2024-03-25T19:35:30.804 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.804+0000 7ffafffff640 1 -- 172.21.15.111:0/1855665229 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb180ac320 con 0x7ffb18142a20 2024-03-25T19:35:30.804 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.804+0000 7ffafffff640 1 -- 172.21.15.111:0/1855665229 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ffb10001b20 con 0x7ffb18142a20 2024-03-25T19:35:30.804 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.804+0000 7ffafffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:30.804 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.804+0000 7ffafffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:30.804 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.804+0000 7ffafffff640 10 monclient(hunting): dump: 2024-03-25T19:35:30.804 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:30.804 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:30.804 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.804 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.804 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:30.804 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:30.804 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:30.804 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:30.804 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:30.805 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:30.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffafffff640 1 -- 172.21.15.111:0/1855665229 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3059545519 0 0) 0x7ffb180adea0 con 0x7ffb180a96b0 2024-03-25T19:35:30.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffafffff640 1 -- 172.21.15.111:0/1855665229 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb04004eb0 con 0x7ffb180a96b0 2024-03-25T19:35:30.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffafffff640 1 -- 172.21.15.111:0/1855665229 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ffb0c001950 con 0x7ffb180a96b0 2024-03-25T19:35:30.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffafffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:30.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffafffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:30.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffafffff640 10 monclient(hunting): dump: 2024-03-25T19:35:30.808 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:30.808 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:30.808 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.808 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.808 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:30.808 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:30.808 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:30.808 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:30.808 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:30.808 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:30.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffafffff640 1 -- 172.21.15.111:0/1855665229 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 325349603 0 0) 0x7ffb180ae180 con 0x7ffb180b4700 2024-03-25T19:35:30.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffafffff640 1 -- 172.21.15.111:0/1855665229 >> v1:172.21.15.44:6790/0 conn(0x7ffb180a96b0 legacy=0x7ffb1813aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:30.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffafffff640 1 -- 172.21.15.111:0/1855665229 >> v1:172.21.15.44:6789/0 conn(0x7ffb18142a20 legacy=0x7ffb18135330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:30.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffafffff640 10 monclient: _finish_hunting 0 2024-03-25T19:35:30.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffafffff640 1 monclient: found mon.b 2024-03-25T19:35:30.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffafffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:30.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffafffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:30.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffafffff640 1 -- 172.21.15.111:0/1855665229 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb180ae460 con 0x7ffb180b4700 2024-03-25T19:35:30.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffafffff640 10 monclient: _finish_auth 0 2024-03-25T19:35:30.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffafffff640 10 monclient: _check_auth_tickets 2024-03-25T19:35:30.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffafffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:30.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffb1dad3640 5 monclient: authenticate success, global_id 4144 2024-03-25T19:35:30.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffb1dad3640 10 monclient: _renew_subs 2024-03-25T19:35:30.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffb1dad3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:30.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffb1dad3640 1 -- 172.21.15.111:0/1855665229 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb18135a40 con 0x7ffb180b4700 2024-03-25T19:35:30.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffb1dad3640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:30.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffb1dad3640 10 monclient: _renew_subs 2024-03-25T19:35:30.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffb1dad3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:30.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffb1dad3640 1 -- 172.21.15.111:0/1855665229 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1813b170 con 0x7ffb180b4700 2024-03-25T19:35:30.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.807+0000 7ffb1dad3640 1 librados: init done 2024-03-25T19:35:30.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.808+0000 7ffb1dad3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:30.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.808+0000 7ffb1dad3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:30.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.808+0000 7ffb1dad3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:30.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.808+0000 7ffb1dad3640 1 -- 172.21.15.111:0/1855665229 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb1804f730 con 0x7ffb180b4700 2024-03-25T19:35:30.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.914+0000 7ffafffff640 1 -- 172.21.15.111:0/1855665229 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb080038a0 con 0x7ffb180b4700 2024-03-25T19:35:30.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.914+0000 7ffafffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:30.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.914+0000 7ffafffff640 1 -- 172.21.15.111:0/1855665229 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ffb08003bb0 con 0x7ffb180b4700 2024-03-25T19:35:30.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.914+0000 7ffafffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:30.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.914+0000 7ffafffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:30.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.914+0000 7ffafffff640 10 monclient: dump: 2024-03-25T19:35:30.915 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:30.915 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:30.915 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.915 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:30.915 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:30.915 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:30.915 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:30.915 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:30.915 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:30.915 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:30.916 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.914+0000 7ffafffff640 1 -- 172.21.15.111:0/1855665229 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ffb080040a0 con 0x7ffb180b4700 2024-03-25T19:35:30.916 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.914+0000 7ffafffff640 1 -- 172.21.15.111:0/1855665229 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 1938606001 0 0) 0x7ffb080056b0 con 0x7ffb180b4700 2024-03-25T19:35:30.916 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.914+0000 7ffb157fa640 4 set_mon_vals no callback set 2024-03-25T19:35:30.916 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.914+0000 7ffafffff640 10 client.4144.objecter ms_dispatch 0x7ffb18000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T19:35:30.916 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.914+0000 7ffafffff640 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T19:35:30.916 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.914+0000 7ffafffff640 3 client.4144.objecter handle_osd_map decoding full epoch 7 2024-03-25T19:35:30.916 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.914+0000 7ffafffff640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-25T19:35:30.916 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.914+0000 7ffafffff640 1 -- 172.21.15.111:0/1855665229 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ffb08005af0 con 0x7ffb180b4700 2024-03-25T19:35:30.916 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.914+0000 7ffafffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:30.916 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.914+0000 7ffafffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:30.986 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.987+0000 7ffb1dad3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ecd09f68-7386-4523-bf9f-f14877e2e746", "id": 5}] 2024-03-25T19:35:30.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.987+0000 7ffb1dad3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ecd09f68-7386-4523-bf9f-f14877e2e746", "id": 5}] 2024-03-25T19:35:30.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.987+0000 7ffb1dad3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:30.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:30.987+0000 7ffb1dad3640 1 -- 172.21.15.111:0/1855665229 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ecd09f68-7386-4523-bf9f-f14877e2e746", "id": 5} v 0) v1 -- 0x7ffb180b1d30 con 0x7ffb180b4700 2024-03-25T19:35:31.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.042+0000 7ffafffff640 1 -- 172.21.15.111:0/1855665229 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ecd09f68-7386-4523-bf9f-f14877e2e746", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 794822422 0 1988412176) 0x7ffb08005130 con 0x7ffb180b4700 2024-03-25T19:35:31.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.042+0000 7ffafffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ecd09f68-7386-4523-bf9f-f14877e2e746", "id": 5}] 2024-03-25T19:35:31.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.042+0000 7ffafffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:31.042 INFO:teuthology.orchestra.run.smithi111.stdout:5 2024-03-25T19:35:31.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.043+0000 7ffb1dad3640 10 librados: watch_flush enter 2024-03-25T19:35:31.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.043+0000 7ffb1dad3640 10 librados: watch_flush exit 2024-03-25T19:35:31.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.043+0000 7ffb1dad3640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-25T19:35:31.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.043+0000 7ffb1dad3640 10 client.4144.objecter successfully canceled tick 2024-03-25T19:35:31.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.043+0000 7ffb1dad3640 10 monclient: shutdown 2024-03-25T19:35:31.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.043+0000 7ffb1dad3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:31.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.043+0000 7ffb1dad3640 1 -- 172.21.15.111:0/1855665229 >> v1:172.21.15.111:6789/0 conn(0x7ffb180b4700 legacy=0x7ffb18134c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:31.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.043+0000 7ffb1dad3640 1 -- 172.21.15.111:0/1855665229 shutdown_connections 2024-03-25T19:35:31.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.043+0000 7ffb1dad3640 1 -- 172.21.15.111:0/1855665229 >> 172.21.15.111:0/1855665229 conn(0x7ffb180668e0 msgr2=0x7ffb1813f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:31.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.043+0000 7ffb1dad3640 1 -- 172.21.15.111:0/1855665229 shutdown_connections 2024-03-25T19:35:31.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.044+0000 7ffb1dad3640 1 -- 172.21.15.111:0/1855665229 wait complete. 2024-03-25T19:35:31.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.044+0000 7ffb1dad3640 1 librados: shutdown 2024-03-25T19:35:31.055 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new 0c107cc3-7f87-4aa3-b3ea-619f471ae849 6 2024-03-25T19:35:31.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.155+0000 7f2c4a715640 10 monclient: get_monmap_and_config 2024-03-25T19:35:31.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.155+0000 7f2c4a715640 10 monclient: build_initial_monmap 2024-03-25T19:35:31.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.155+0000 7f2c4a715640 10 monclient: monmap: 2024-03-25T19:35:31.155 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T19:35:31.155 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:31.155 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:35:31.156942+0000 2024-03-25T19:35:31.155 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:35:31.156942+0000 2024-03-25T19:35:31.156 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:31.156 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:31.156 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:31.156 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:35:31.156 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:35:31.156 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:31.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.156+0000 7f2c4a715640 1 Processor -- start 2024-03-25T19:35:31.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.156+0000 7f2c4a715640 1 -- start start 2024-03-25T19:35:31.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.156+0000 7f2c4a715640 10 monclient: init 2024-03-25T19:35:31.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.157+0000 7f2c4a715640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:31.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.157+0000 7f2c4a715640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:31.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.157+0000 7f2c4a715640 10 monclient(hunting): picked mon.noname-b con 0x7f2c440b4700 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:31.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.157+0000 7f2c4a715640 10 monclient(hunting): picked mon.noname-c con 0x7f2c44142a20 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:31.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.157+0000 7f2c4a715640 10 monclient(hunting): picked mon.noname-a con 0x7f2c440a96b0 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:31.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.157+0000 7f2c4a715640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c440a9d60 con 0x7f2c440a96b0 2024-03-25T19:35:31.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.157+0000 7f2c4a715640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c4413ff10 con 0x7f2c44142a20 2024-03-25T19:35:31.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.157+0000 7f2c4a715640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c440aa1f0 con 0x7f2c440b4700 2024-03-25T19:35:31.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.157+0000 7f2c4a715640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:31.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.157+0000 7f2c4a715640 10 monclient(hunting): authenticate will time out at 5352.293066s 2024-03-25T19:35:31.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.157+0000 7f2c43fff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f2c440b4700 0x7f2c44142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.111:42478/0 (socket says 172.21.15.111:42478) 2024-03-25T19:35:31.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.157+0000 7f2c43fff640 1 -- 172.21.15.111:0/731606696 learned_addr learned my addr 172.21.15.111:0/731606696 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T19:35:31.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.160+0000 7f2c42ffd640 1 -- 172.21.15.111:0/731606696 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1826000861 0 0) 0x7f2c4413ff10 con 0x7f2c44142a20 2024-03-25T19:35:31.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.160+0000 7f2c42ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:31.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.160+0000 7f2c42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:31.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.160+0000 7f2c42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:31.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.160+0000 7f2c42ffd640 10 monclient(hunting): my global_id is 4147 2024-03-25T19:35:31.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.160+0000 7f2c42ffd640 1 -- 172.21.15.111:0/731606696 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c2c003520 con 0x7f2c44142a20 2024-03-25T19:35:31.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.160+0000 7f2c42ffd640 1 -- 172.21.15.111:0/731606696 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578756278 0 0) 0x7f2c440aa1f0 con 0x7f2c440b4700 2024-03-25T19:35:31.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.160+0000 7f2c42ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:31.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.160+0000 7f2c42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:31.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.160+0000 7f2c42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:31.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.160+0000 7f2c42ffd640 10 monclient(hunting): my global_id is 4151 2024-03-25T19:35:31.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.160+0000 7f2c42ffd640 1 -- 172.21.15.111:0/731606696 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c4413ff10 con 0x7f2c440b4700 2024-03-25T19:35:31.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.160+0000 7f2c42ffd640 1 -- 172.21.15.111:0/731606696 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921037065 0 0) 0x7f2c440a9d60 con 0x7f2c440a96b0 2024-03-25T19:35:31.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.160+0000 7f2c42ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:31.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.160+0000 7f2c42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:31.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.160+0000 7f2c42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:31.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.160+0000 7f2c42ffd640 10 monclient(hunting): my global_id is 4152 2024-03-25T19:35:31.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.160+0000 7f2c42ffd640 1 -- 172.21.15.111:0/731606696 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c440aa1f0 con 0x7f2c440a96b0 2024-03-25T19:35:31.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.163+0000 7f2c42ffd640 1 -- 172.21.15.111:0/731606696 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2782982312 0 0) 0x7f2c2c003520 con 0x7f2c44142a20 2024-03-25T19:35:31.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.163+0000 7f2c42ffd640 1 -- 172.21.15.111:0/731606696 >> v1:172.21.15.44:6790/0 conn(0x7f2c440b4700 legacy=0x7f2c44142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:31.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.163+0000 7f2c42ffd640 1 -- 172.21.15.111:0/731606696 >> v1:172.21.15.44:6789/0 conn(0x7f2c440a96b0 legacy=0x7f2c4413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:31.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.163+0000 7f2c42ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:35:31.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.163+0000 7f2c42ffd640 1 monclient: found mon.noname-c 2024-03-25T19:35:31.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.163+0000 7f2c42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:31.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.163+0000 7f2c42ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:35:31.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.163+0000 7f2c42ffd640 1 -- 172.21.15.111:0/731606696 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c440b4ae0 con 0x7f2c44142a20 2024-03-25T19:35:31.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.163+0000 7f2c42ffd640 10 monclient: _finish_auth 0 2024-03-25T19:35:31.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.163+0000 7f2c42ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:35:31.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.163+0000 7f2c42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:31.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.163+0000 7f2c4a715640 5 monclient: authenticate success, global_id 4147 2024-03-25T19:35:31.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.163+0000 7f2c42ffd640 1 -- 172.21.15.111:0/731606696 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f2c34001690 con 0x7f2c44142a20 2024-03-25T19:35:31.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.163+0000 7f2c42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:31.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.163+0000 7f2c42ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:31.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.163+0000 7f2c42ffd640 10 monclient: dump: 2024-03-25T19:35:31.164 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:31.164 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:31.164 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:31.164 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:31.164 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:31.164 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:31.165 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:31.165 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:31.165 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:31.165 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:31.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.163+0000 7f2c4a715640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:31.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.164+0000 7f2c42ffd640 1 -- 172.21.15.111:0/731606696 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c340017f0 con 0x7f2c44142a20 2024-03-25T19:35:31.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.164+0000 7f2c42ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:31.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.164+0000 7f2c4a715640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:31.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.164+0000 7f2c4a715640 4 set_mon_vals no callback set 2024-03-25T19:35:31.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.164+0000 7f2c42ffd640 1 -- 172.21.15.111:0/731606696 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f2c34001c00 con 0x7f2c44142a20 2024-03-25T19:35:31.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.164+0000 7f2c42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:31.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.164+0000 7f2c42ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:31.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.164+0000 7f2c42ffd640 10 monclient: dump: 2024-03-25T19:35:31.168 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:31.169 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:31.169 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:31.169 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:31.169 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:31.169 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:31.169 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:31.169 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:31.169 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:31.169 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:31.169 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.165+0000 7f2c4a715640 10 monclient: shutdown 2024-03-25T19:35:31.169 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.165+0000 7f2c4a715640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:31.169 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.165+0000 7f2c4a715640 1 -- 172.21.15.111:0/731606696 >> v1:172.21.15.111:6789/0 conn(0x7f2c44142a20 legacy=0x7f2c4413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:31.169 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.165+0000 7f2c4a715640 1 -- 172.21.15.111:0/731606696 shutdown_connections 2024-03-25T19:35:31.169 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.165+0000 7f2c4a715640 1 -- 172.21.15.111:0/731606696 >> 172.21.15.111:0/731606696 conn(0x7f2c440668e0 msgr2=0x7f2c44066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:31.170 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.165+0000 7f2c4a715640 1 -- 172.21.15.111:0/731606696 shutdown_connections 2024-03-25T19:35:31.170 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.165+0000 7f2c4a715640 1 -- 172.21.15.111:0/731606696 wait complete. 2024-03-25T19:35:31.170 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.166+0000 7f2c4a715640 10 monclient: build_initial_monmap 2024-03-25T19:35:31.170 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.166+0000 7f2c4a715640 10 monclient: monmap: 2024-03-25T19:35:31.170 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T19:35:31.170 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:31.170 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 0.000000 2024-03-25T19:35:31.170 INFO:teuthology.orchestra.run.smithi111.stderr:created 0.000000 2024-03-25T19:35:31.170 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:31.170 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 1 2024-03-25T19:35:31.170 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:31.170 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:35:31.170 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:35:31.171 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:31.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.166+0000 7f2c4a715640 1 librados: starting msgr at 2024-03-25T19:35:31.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.166+0000 7f2c4a715640 1 librados: starting objecter 2024-03-25T19:35:31.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.166+0000 7f2c4a715640 1 Processor -- start 2024-03-25T19:35:31.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.166+0000 7f2c4a715640 1 -- start start 2024-03-25T19:35:31.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.166+0000 7f2c4a715640 1 librados: setting wanted keys 2024-03-25T19:35:31.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.166+0000 7f2c4a715640 1 librados: calling monclient init 2024-03-25T19:35:31.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.166+0000 7f2c4a715640 10 monclient: init 2024-03-25T19:35:31.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.167+0000 7f2c4a715640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:31.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.167+0000 7f2c4a715640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:31.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.167+0000 7f2c4a715640 10 monclient(hunting): picked mon.noname-b con 0x7f2c440a96b0 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:31.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.167+0000 7f2c4a715640 10 monclient(hunting): picked mon.noname-a con 0x7f2c440b4700 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:31.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.167+0000 7f2c4a715640 10 monclient(hunting): picked mon.noname-c con 0x7f2c44142a20 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:31.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.167+0000 7f2c4a715640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c440abb60 con 0x7f2c440b4700 2024-03-25T19:35:31.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.167+0000 7f2c4a715640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c440abdb0 con 0x7f2c440a96b0 2024-03-25T19:35:31.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.167+0000 7f2c4a715640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c4414e850 con 0x7f2c44142a20 2024-03-25T19:35:31.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.167+0000 7f2c4a715640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:31.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.167+0000 7f2c4a715640 10 monclient(hunting): authenticate will time out at 5352.303014s 2024-03-25T19:35:31.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.167+0000 7f2c48cb2640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f2c44142a20 0x7f2c441e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.111:42480/0 (socket says 172.21.15.111:42480) 2024-03-25T19:35:31.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.167+0000 7f2c48cb2640 1 -- 172.21.15.111:0/1540801897 learned_addr learned my addr 172.21.15.111:0/1540801897 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T19:35:31.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.167+0000 7f2c40ff9640 10 client.?.objecter ms_handle_connect 0x7f2c44142a20 2024-03-25T19:35:31.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.167+0000 7f2c40ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:31.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.168+0000 7f2c40ff9640 10 client.?.objecter ms_handle_connect 0x7f2c440a96b0 2024-03-25T19:35:31.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.168+0000 7f2c40ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:31.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.168+0000 7f2c40ff9640 10 client.?.objecter ms_handle_connect 0x7f2c440b4700 2024-03-25T19:35:31.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.168+0000 7f2c40ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:31.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.169+0000 7f2c40ff9640 1 -- 172.21.15.111:0/1540801897 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688658323 0 0) 0x7f2c4414e850 con 0x7f2c44142a20 2024-03-25T19:35:31.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.170+0000 7f2c40ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:31.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.170+0000 7f2c40ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:31.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.170+0000 7f2c40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:31.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.170+0000 7f2c40ff9640 10 monclient(hunting): my global_id is 4154 2024-03-25T19:35:31.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.170+0000 7f2c40ff9640 1 -- 172.21.15.111:0/1540801897 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c30004eb0 con 0x7f2c44142a20 2024-03-25T19:35:31.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.170+0000 7f2c40ff9640 1 -- 172.21.15.111:0/1540801897 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675285816 0 0) 0x7f2c440abdb0 con 0x7f2c440a96b0 2024-03-25T19:35:31.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.170+0000 7f2c40ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:31.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.170+0000 7f2c40ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:31.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.170+0000 7f2c40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:31.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.170+0000 7f2c40ff9640 10 monclient(hunting): my global_id is 4150 2024-03-25T19:35:31.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.170+0000 7f2c40ff9640 1 -- 172.21.15.111:0/1540801897 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c4414e850 con 0x7f2c440a96b0 2024-03-25T19:35:31.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.170+0000 7f2c40ff9640 1 -- 172.21.15.111:0/1540801897 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2767178214 0 0) 0x7f2c440abb60 con 0x7f2c440b4700 2024-03-25T19:35:31.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.170+0000 7f2c40ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:31.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.170+0000 7f2c40ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:31.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.170+0000 7f2c40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:31.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.170+0000 7f2c40ff9640 10 monclient(hunting): my global_id is 4155 2024-03-25T19:35:31.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.170+0000 7f2c40ff9640 1 -- 172.21.15.111:0/1540801897 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c440abdb0 con 0x7f2c440b4700 2024-03-25T19:35:31.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.174+0000 7f2c40ff9640 1 -- 172.21.15.111:0/1540801897 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3509568033 0 0) 0x7f2c4414e850 con 0x7f2c440a96b0 2024-03-25T19:35:31.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.174+0000 7f2c40ff9640 1 -- 172.21.15.111:0/1540801897 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c440abb60 con 0x7f2c440a96b0 2024-03-25T19:35:31.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.174+0000 7f2c40ff9640 1 -- 172.21.15.111:0/1540801897 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f2c38001970 con 0x7f2c440a96b0 2024-03-25T19:35:31.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.174+0000 7f2c40ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:31.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.174+0000 7f2c40ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:31.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.174+0000 7f2c40ff9640 10 monclient(hunting): dump: 2024-03-25T19:35:31.176 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:31.176 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:31.176 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:31.176 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:31.176 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:31.177 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:31.177 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:31.177 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:31.177 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:31.177 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:31.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.174+0000 7f2c40ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:31.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.174+0000 7f2c40ff9640 1 -- 172.21.15.111:0/1540801897 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3998771178 0 0) 0x7f2c30004eb0 con 0x7f2c44142a20 2024-03-25T19:35:31.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.174+0000 7f2c40ff9640 1 -- 172.21.15.111:0/1540801897 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c4414e850 con 0x7f2c44142a20 2024-03-25T19:35:31.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.174+0000 7f2c40ff9640 1 -- 172.21.15.111:0/1540801897 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f2c3c001d10 con 0x7f2c44142a20 2024-03-25T19:35:31.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.174+0000 7f2c40ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:31.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.174+0000 7f2c40ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:31.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.174+0000 7f2c40ff9640 10 monclient(hunting): dump: 2024-03-25T19:35:31.177 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:31.177 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:31.177 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:31.178 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:31.178 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:31.178 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:31.178 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:31.178 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:31.178 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:31.178 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:31.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.175+0000 7f2c40ff9640 1 -- 172.21.15.111:0/1540801897 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1222995210 0 0) 0x7f2c440abdb0 con 0x7f2c440b4700 2024-03-25T19:35:31.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.175+0000 7f2c40ff9640 1 -- 172.21.15.111:0/1540801897 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c30004eb0 con 0x7f2c440b4700 2024-03-25T19:35:31.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.175+0000 7f2c40ff9640 1 -- 172.21.15.111:0/1540801897 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f2c34001b00 con 0x7f2c440b4700 2024-03-25T19:35:31.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.175+0000 7f2c40ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:31.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.175+0000 7f2c40ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:31.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.175+0000 7f2c40ff9640 10 monclient(hunting): dump: 2024-03-25T19:35:31.179 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:31.179 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:31.179 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:31.179 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:31.179 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:31.179 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:31.179 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:31.179 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:31.179 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:31.179 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:31.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.177+0000 7f2c40ff9640 1 -- 172.21.15.111:0/1540801897 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 646936942 0 0) 0x7f2c440abb60 con 0x7f2c440a96b0 2024-03-25T19:35:31.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.177+0000 7f2c40ff9640 1 -- 172.21.15.111:0/1540801897 >> v1:172.21.15.44:6790/0 conn(0x7f2c44142a20 legacy=0x7f2c441e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:31.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.177+0000 7f2c40ff9640 1 -- 172.21.15.111:0/1540801897 >> v1:172.21.15.44:6789/0 conn(0x7f2c440b4700 legacy=0x7f2c4413bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:31.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.177+0000 7f2c40ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:35:31.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.177+0000 7f2c40ff9640 1 monclient: found mon.b 2024-03-25T19:35:31.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.177+0000 7f2c40ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:31.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.177+0000 7f2c40ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:31.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.177+0000 7f2c40ff9640 1 -- 172.21.15.111:0/1540801897 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c44139be0 con 0x7f2c440a96b0 2024-03-25T19:35:31.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.177+0000 7f2c40ff9640 10 monclient: _finish_auth 0 2024-03-25T19:35:31.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.177+0000 7f2c40ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:35:31.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.177+0000 7f2c40ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:31.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.177+0000 7f2c4a715640 5 monclient: authenticate success, global_id 4150 2024-03-25T19:35:31.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.177+0000 7f2c4a715640 10 monclient: _renew_subs 2024-03-25T19:35:31.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.177+0000 7f2c4a715640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:31.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.177+0000 7f2c4a715640 1 -- 172.21.15.111:0/1540801897 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c4413c9c0 con 0x7f2c440a96b0 2024-03-25T19:35:31.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.177+0000 7f2c4a715640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:31.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.177+0000 7f2c4a715640 10 monclient: _renew_subs 2024-03-25T19:35:31.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.177+0000 7f2c4a715640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:31.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.177+0000 7f2c4a715640 1 -- 172.21.15.111:0/1540801897 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c440ac690 con 0x7f2c440a96b0 2024-03-25T19:35:31.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.177+0000 7f2c4a715640 1 librados: init done 2024-03-25T19:35:31.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.178+0000 7f2c4a715640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:31.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.178+0000 7f2c4a715640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:31.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.178+0000 7f2c4a715640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:31.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.178+0000 7f2c4a715640 1 -- 172.21.15.111:0/1540801897 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c440b76e0 con 0x7f2c440a96b0 2024-03-25T19:35:31.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.179+0000 7f2c40ff9640 1 -- 172.21.15.111:0/1540801897 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c380047e0 con 0x7f2c440a96b0 2024-03-25T19:35:31.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.179+0000 7f2c40ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:31.183 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.179+0000 7f2c427fc640 4 set_mon_vals no callback set 2024-03-25T19:35:31.280 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.279+0000 7f2c40ff9640 1 -- 172.21.15.111:0/1540801897 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f2c38004af0 con 0x7f2c440a96b0 2024-03-25T19:35:31.280 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.279+0000 7f2c40ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:31.280 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.279+0000 7f2c40ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:31.280 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.279+0000 7f2c40ff9640 10 monclient: dump: 2024-03-25T19:35:31.280 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:31.280 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:31.280 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:31.280 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:31.280 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:31.280 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:31.280 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:31.280 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:31.280 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:31.280 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:31.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.279+0000 7f2c40ff9640 1 -- 172.21.15.111:0/1540801897 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2c38004fe0 con 0x7f2c440a96b0 2024-03-25T19:35:31.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.279+0000 7f2c40ff9640 1 -- 172.21.15.111:0/1540801897 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2760849767 0 0) 0x7f2c38006810 con 0x7f2c440a96b0 2024-03-25T19:35:31.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.279+0000 7f2c40ff9640 10 client.4150.objecter ms_dispatch 0x7f2c44000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T19:35:31.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.279+0000 7f2c40ff9640 3 client.4150.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T19:35:31.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.279+0000 7f2c40ff9640 3 client.4150.objecter handle_osd_map decoding full epoch 8 2024-03-25T19:35:31.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.279+0000 7f2c40ff9640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-25T19:35:31.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.279+0000 7f2c40ff9640 1 -- 172.21.15.111:0/1540801897 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2c38006bf0 con 0x7f2c440a96b0 2024-03-25T19:35:31.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.280+0000 7f2c40ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:31.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.280+0000 7f2c40ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:31.330 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:31.331+0000 7f882064b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:35:31.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.354+0000 7f2c4a715640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0c107cc3-7f87-4aa3-b3ea-619f471ae849", "id": 6}] 2024-03-25T19:35:31.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.354+0000 7f2c4a715640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0c107cc3-7f87-4aa3-b3ea-619f471ae849", "id": 6}] 2024-03-25T19:35:31.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.354+0000 7f2c4a715640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:31.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.354+0000 7f2c4a715640 1 -- 172.21.15.111:0/1540801897 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd new", "uuid": "0c107cc3-7f87-4aa3-b3ea-619f471ae849", "id": 6} v 0) v1 -- 0x7f2c4404f730 con 0x7f2c440a96b0 2024-03-25T19:35:31.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.436+0000 7f2c40ff9640 1 -- 172.21.15.111:0/1540801897 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0c107cc3-7f87-4aa3-b3ea-619f471ae849", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 4188595070 0 1113811849) 0x7f2c380062f0 con 0x7f2c440a96b0 2024-03-25T19:35:31.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.436+0000 7f2c40ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0c107cc3-7f87-4aa3-b3ea-619f471ae849", "id": 6}] 2024-03-25T19:35:31.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.436+0000 7f2c40ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:31.436 INFO:teuthology.orchestra.run.smithi111.stdout:6 2024-03-25T19:35:31.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.437+0000 7f2c4a715640 10 librados: watch_flush enter 2024-03-25T19:35:31.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.437+0000 7f2c4a715640 10 librados: watch_flush exit 2024-03-25T19:35:31.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.437+0000 7f2c4a715640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-25T19:35:31.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.437+0000 7f2c4a715640 10 client.4150.objecter successfully canceled tick 2024-03-25T19:35:31.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.437+0000 7f2c4a715640 10 monclient: shutdown 2024-03-25T19:35:31.438 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.437+0000 7f2c4a715640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:31.438 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.437+0000 7f2c4a715640 1 -- 172.21.15.111:0/1540801897 >> v1:172.21.15.111:6789/0 conn(0x7f2c440a96b0 legacy=0x7f2c4413b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:31.438 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.437+0000 7f2c4a715640 1 -- 172.21.15.111:0/1540801897 shutdown_connections 2024-03-25T19:35:31.438 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.437+0000 7f2c4a715640 1 -- 172.21.15.111:0/1540801897 >> 172.21.15.111:0/1540801897 conn(0x7f2c440668e0 msgr2=0x7f2c440b2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:31.438 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.437+0000 7f2c4a715640 1 -- 172.21.15.111:0/1540801897 shutdown_connections 2024-03-25T19:35:31.438 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.437+0000 7f2c4a715640 1 -- 172.21.15.111:0/1540801897 wait complete. 2024-03-25T19:35:31.438 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.437+0000 7f2c4a715640 1 librados: shutdown 2024-03-25T19:35:31.447 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new f504270b-9a52-449e-8e05-e65bde3085cc 7 2024-03-25T19:35:31.517 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T19:35:31.517+0000 7f54ee4c7040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:35:31.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.546+0000 7f27b608f640 10 monclient: get_monmap_and_config 2024-03-25T19:35:31.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.546+0000 7f27b608f640 10 monclient: build_initial_monmap 2024-03-25T19:35:31.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.546+0000 7f27b608f640 10 monclient: monmap: 2024-03-25T19:35:31.546 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T19:35:31.546 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:31.546 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:35:31.547736+0000 2024-03-25T19:35:31.547 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:35:31.547736+0000 2024-03-25T19:35:31.547 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:31.547 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:31.547 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:31.547 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:35:31.547 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:35:31.547 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:31.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.547+0000 7f27b608f640 1 Processor -- start 2024-03-25T19:35:31.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.547+0000 7f27b608f640 1 -- start start 2024-03-25T19:35:31.549 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.547+0000 7f27b608f640 10 monclient: init 2024-03-25T19:35:31.549 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.548+0000 7f27b608f640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:31.549 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.548+0000 7f27b608f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:31.549 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.548+0000 7f27b608f640 10 monclient(hunting): picked mon.noname-c con 0x7f27b00b4700 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:31.549 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.548+0000 7f27b608f640 10 monclient(hunting): picked mon.noname-a con 0x7f27b0142a20 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:31.549 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.548+0000 7f27b608f640 10 monclient(hunting): picked mon.noname-b con 0x7f27b00a96b0 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:31.549 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.548+0000 7f27b608f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b00a9d60 con 0x7f27b0142a20 2024-03-25T19:35:31.549 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.548+0000 7f27b608f640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b013ff10 con 0x7f27b00b4700 2024-03-25T19:35:31.549 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.548+0000 7f27b608f640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b00aa1f0 con 0x7f27b00a96b0 2024-03-25T19:35:31.549 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.548+0000 7f27b608f640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:31.549 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.548+0000 7f27b608f640 10 monclient(hunting): authenticate will time out at 5352.684004s 2024-03-25T19:35:31.549 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.548+0000 7f27af7fe640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f27b00b4700 0x7f27b0142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:47358/0 (socket says 172.21.15.111:47358) 2024-03-25T19:35:31.549 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.548+0000 7f27af7fe640 1 -- 172.21.15.111:0/3979398076 learned_addr learned my addr 172.21.15.111:0/3979398076 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T19:35:31.550 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.551+0000 7f27aeffd640 1 -- 172.21.15.111:0/3979398076 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3949698361 0 0) 0x7f27b013ff10 con 0x7f27b00b4700 2024-03-25T19:35:31.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.551+0000 7f27aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:31.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.551+0000 7f27aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:31.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.551+0000 7f27aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:31.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.551+0000 7f27aeffd640 10 monclient(hunting): my global_id is 4153 2024-03-25T19:35:31.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.551+0000 7f27aeffd640 1 -- 172.21.15.111:0/3979398076 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f279c003520 con 0x7f27b00b4700 2024-03-25T19:35:31.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.551+0000 7f27aeffd640 1 -- 172.21.15.111:0/3979398076 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1976346400 0 0) 0x7f27b00aa1f0 con 0x7f27b00a96b0 2024-03-25T19:35:31.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.551+0000 7f27aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:31.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.551+0000 7f27aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:31.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.551+0000 7f27aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:31.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.551+0000 7f27aeffd640 10 monclient(hunting): my global_id is 4157 2024-03-25T19:35:31.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.551+0000 7f27aeffd640 1 -- 172.21.15.111:0/3979398076 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27b013ff10 con 0x7f27b00a96b0 2024-03-25T19:35:31.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.552+0000 7f27aeffd640 1 -- 172.21.15.111:0/3979398076 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966532435 0 0) 0x7f27b00a9d60 con 0x7f27b0142a20 2024-03-25T19:35:31.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.552+0000 7f27aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:31.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.552+0000 7f27aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:31.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.552+0000 7f27aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:31.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.552+0000 7f27aeffd640 10 monclient(hunting): my global_id is 4158 2024-03-25T19:35:31.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.552+0000 7f27aeffd640 1 -- 172.21.15.111:0/3979398076 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27b00aa1f0 con 0x7f27b0142a20 2024-03-25T19:35:31.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.554+0000 7f27aeffd640 1 -- 172.21.15.111:0/3979398076 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1533849005 0 0) 0x7f279c003520 con 0x7f27b00b4700 2024-03-25T19:35:31.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.554+0000 7f27aeffd640 1 -- 172.21.15.111:0/3979398076 >> v1:172.21.15.44:6790/0 conn(0x7f27b00a96b0 legacy=0x7f27b013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:31.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.554+0000 7f27aeffd640 1 -- 172.21.15.111:0/3979398076 >> v1:172.21.15.44:6789/0 conn(0x7f27b0142a20 legacy=0x7f27b013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:31.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.554+0000 7f27aeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:35:31.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.554+0000 7f27aeffd640 1 monclient: found mon.noname-c 2024-03-25T19:35:31.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.554+0000 7f27aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:31.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.554+0000 7f27aeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:35:31.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.554+0000 7f27aeffd640 1 -- 172.21.15.111:0/3979398076 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b00b4ae0 con 0x7f27b00b4700 2024-03-25T19:35:31.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.554+0000 7f27aeffd640 10 monclient: _finish_auth 0 2024-03-25T19:35:31.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.554+0000 7f27aeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:35:31.555 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.554+0000 7f27aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:31.555 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.554+0000 7f27aeffd640 1 -- 172.21.15.111:0/3979398076 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f27a0001690 con 0x7f27b00b4700 2024-03-25T19:35:31.555 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.554+0000 7f27b608f640 5 monclient: authenticate success, global_id 4153 2024-03-25T19:35:31.555 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.554+0000 7f27b608f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:31.555 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.554+0000 7f27aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:31.555 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.554+0000 7f27aeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:31.555 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.554+0000 7f27aeffd640 10 monclient: dump: 2024-03-25T19:35:31.555 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:31.555 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:31.555 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:31.555 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:31.555 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:31.555 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:31.555 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:31.555 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:31.556 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:31.556 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:31.556 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.554+0000 7f27b608f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:31.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.555+0000 7f27aeffd640 1 -- 172.21.15.111:0/3979398076 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27a00017f0 con 0x7f27b00b4700 2024-03-25T19:35:31.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.556+0000 7f27aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:31.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.556+0000 7f27aeffd640 1 -- 172.21.15.111:0/3979398076 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f27a0001c00 con 0x7f27b00b4700 2024-03-25T19:35:31.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.556+0000 7f27aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:31.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.556+0000 7f27aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:31.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.556+0000 7f27aeffd640 10 monclient: dump: 2024-03-25T19:35:31.559 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:31.559 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:31.559 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:31.560 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.556+0000 7f27b608f640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.556+0000 7f27b608f640 4 set_mon_vals no callback set 2024-03-25T19:35:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.556+0000 7f27b608f640 10 monclient: shutdown 2024-03-25T19:35:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.556+0000 7f27b608f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.556+0000 7f27b608f640 1 -- 172.21.15.111:0/3979398076 >> v1:172.21.15.111:6789/0 conn(0x7f27b00b4700 legacy=0x7f27b0142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.556+0000 7f27b608f640 1 -- 172.21.15.111:0/3979398076 shutdown_connections 2024-03-25T19:35:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.556+0000 7f27b608f640 1 -- 172.21.15.111:0/3979398076 >> 172.21.15.111:0/3979398076 conn(0x7f27b00668e0 msgr2=0x7f27b0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.556+0000 7f27b608f640 1 -- 172.21.15.111:0/3979398076 shutdown_connections 2024-03-25T19:35:31.561 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.556+0000 7f27b608f640 1 -- 172.21.15.111:0/3979398076 wait complete. 2024-03-25T19:35:31.561 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.557+0000 7f27b608f640 10 monclient: build_initial_monmap 2024-03-25T19:35:31.561 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.557+0000 7f27b608f640 10 monclient: monmap: 2024-03-25T19:35:31.561 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 0 2024-03-25T19:35:31.561 INFO:teuthology.orchestra.run.smithi111.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:31.561 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 0.000000 2024-03-25T19:35:31.561 INFO:teuthology.orchestra.run.smithi111.stderr:created 0.000000 2024-03-25T19:35:31.561 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:31.561 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 1 2024-03-25T19:35:31.561 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:31.561 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:35:31.561 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:35:31.561 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:31.561 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.557+0000 7f27b608f640 1 librados: starting msgr at 2024-03-25T19:35:31.561 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.557+0000 7f27b608f640 1 librados: starting objecter 2024-03-25T19:35:31.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.557+0000 7f27b608f640 1 Processor -- start 2024-03-25T19:35:31.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.557+0000 7f27b608f640 1 -- start start 2024-03-25T19:35:31.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.557+0000 7f27b608f640 1 librados: setting wanted keys 2024-03-25T19:35:31.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.557+0000 7f27b608f640 1 librados: calling monclient init 2024-03-25T19:35:31.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.557+0000 7f27b608f640 10 monclient: init 2024-03-25T19:35:31.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.558+0000 7f27b608f640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:31.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.558+0000 7f27b608f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:31.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.558+0000 7f27b608f640 10 monclient(hunting): picked mon.noname-b con 0x7f27b00a96b0 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:31.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.558+0000 7f27b608f640 10 monclient(hunting): picked mon.noname-a con 0x7f27b00b4700 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:31.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.558+0000 7f27b608f640 10 monclient(hunting): picked mon.noname-c con 0x7f27b0142a20 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:31.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.558+0000 7f27b608f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b014eca0 con 0x7f27b00b4700 2024-03-25T19:35:31.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.558+0000 7f27b608f640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b01d74d0 con 0x7f27b00a96b0 2024-03-25T19:35:31.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.558+0000 7f27b608f640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b01d8350 con 0x7f27b0142a20 2024-03-25T19:35:31.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.558+0000 7f27b608f640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:31.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.558+0000 7f27b608f640 10 monclient(hunting): authenticate will time out at 5352.694133s 2024-03-25T19:35:31.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.558+0000 7f27affff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f27b0142a20 0x7f27b01d4780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.111:42498/0 (socket says 172.21.15.111:42498) 2024-03-25T19:35:31.563 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.558+0000 7f27affff640 1 -- 172.21.15.111:0/2724276643 learned_addr learned my addr 172.21.15.111:0/2724276643 (peer_addr_for_me v1:172.21.15.111:0/0) 2024-03-25T19:35:31.563 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.559+0000 7f27acff9640 10 client.?.objecter ms_handle_connect 0x7f27b00a96b0 2024-03-25T19:35:31.563 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.559+0000 7f27acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:31.563 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.559+0000 7f27acff9640 10 client.?.objecter ms_handle_connect 0x7f27b0142a20 2024-03-25T19:35:31.563 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.559+0000 7f27acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:31.563 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.559+0000 7f27acff9640 10 client.?.objecter ms_handle_connect 0x7f27b00b4700 2024-03-25T19:35:31.563 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.559+0000 7f27acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:31.564 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.561+0000 7f27acff9640 1 -- 172.21.15.111:0/2724276643 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281703048 0 0) 0x7f27b01d74d0 con 0x7f27b00a96b0 2024-03-25T19:35:31.564 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.561+0000 7f27acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:31.564 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.561+0000 7f27acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:31.564 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.561+0000 7f27acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:31.564 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.561+0000 7f27acff9640 10 monclient(hunting): my global_id is 4156 2024-03-25T19:35:31.564 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.561+0000 7f27acff9640 1 -- 172.21.15.111:0/2724276643 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2790004eb0 con 0x7f27b00a96b0 2024-03-25T19:35:31.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.561+0000 7f27acff9640 1 -- 172.21.15.111:0/2724276643 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927058418 0 0) 0x7f27b01d8350 con 0x7f27b0142a20 2024-03-25T19:35:31.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.561+0000 7f27acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:31.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.561+0000 7f27acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:31.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.561+0000 7f27acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:31.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.561+0000 7f27acff9640 10 monclient(hunting): my global_id is 4160 2024-03-25T19:35:31.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.561+0000 7f27acff9640 1 -- 172.21.15.111:0/2724276643 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27b01d74d0 con 0x7f27b0142a20 2024-03-25T19:35:31.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.562+0000 7f27acff9640 1 -- 172.21.15.111:0/2724276643 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 381331364 0 0) 0x7f27b00aa1f0 con 0x7f27b00b4700 2024-03-25T19:35:31.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.562+0000 7f27acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:31.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.562+0000 7f27acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:31.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.562+0000 7f27acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:31.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.562+0000 7f27acff9640 10 monclient(hunting): my global_id is 4161 2024-03-25T19:35:31.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.562+0000 7f27acff9640 1 -- 172.21.15.111:0/2724276643 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27b01d8350 con 0x7f27b00b4700 2024-03-25T19:35:31.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.565+0000 7f27acff9640 1 -- 172.21.15.111:0/2724276643 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3290585816 0 0) 0x7f2790004eb0 con 0x7f27b00a96b0 2024-03-25T19:35:31.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.565+0000 7f27acff9640 1 -- 172.21.15.111:0/2724276643 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27b00aa1f0 con 0x7f27b00a96b0 2024-03-25T19:35:31.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.565+0000 7f27acff9640 1 -- 172.21.15.111:0/2724276643 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f27a0001b00 con 0x7f27b00a96b0 2024-03-25T19:35:31.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.565+0000 7f27acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:31.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.565+0000 7f27acff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:31.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.565+0000 7f27acff9640 10 monclient(hunting): dump: 2024-03-25T19:35:31.567 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:31.567 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:31.567 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:31.567 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:31.567 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:31.567 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:31.567 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:31.567 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:31.567 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:31.567 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:31.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.565+0000 7f27acff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:31.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.568+0000 7f27acff9640 1 -- 172.21.15.111:0/2724276643 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1602334941 0 0) 0x7f27b00aa1f0 con 0x7f27b00a96b0 2024-03-25T19:35:31.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.568+0000 7f27acff9640 1 -- 172.21.15.111:0/2724276643 >> v1:172.21.15.44:6790/0 conn(0x7f27b0142a20 legacy=0x7f27b01d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:31.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.568+0000 7f27acff9640 1 -- 172.21.15.111:0/2724276643 >> v1:172.21.15.44:6789/0 conn(0x7f27b00b4700 legacy=0x7f27b00acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:31.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.568+0000 7f27acff9640 10 monclient: _finish_hunting 0 2024-03-25T19:35:31.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.568+0000 7f27acff9640 1 monclient: found mon.b 2024-03-25T19:35:31.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.568+0000 7f27acff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:31.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.568+0000 7f27acff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:31.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.568+0000 7f27acff9640 1 -- 172.21.15.111:0/2724276643 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b00abd90 con 0x7f27b00a96b0 2024-03-25T19:35:31.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.568+0000 7f27acff9640 10 monclient: _finish_auth 0 2024-03-25T19:35:31.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.568+0000 7f27acff9640 10 monclient: _check_auth_tickets 2024-03-25T19:35:31.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.568+0000 7f27acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:31.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.568+0000 7f27b608f640 5 monclient: authenticate success, global_id 4156 2024-03-25T19:35:31.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.568+0000 7f27b608f640 10 monclient: _renew_subs 2024-03-25T19:35:31.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.568+0000 7f27b608f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:31.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.568+0000 7f27b608f640 1 -- 172.21.15.111:0/2724276643 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27b01d7720 con 0x7f27b00a96b0 2024-03-25T19:35:31.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.568+0000 7f27b608f640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:31.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.568+0000 7f27b608f640 10 monclient: _renew_subs 2024-03-25T19:35:31.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.568+0000 7f27b608f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:31.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.568+0000 7f27b608f640 1 -- 172.21.15.111:0/2724276643 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27b00ad240 con 0x7f27b00a96b0 2024-03-25T19:35:31.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.568+0000 7f27b608f640 1 librados: init done 2024-03-25T19:35:31.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.568+0000 7f27b608f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:31.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.568+0000 7f27b608f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:31.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.568+0000 7f27b608f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:31.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.569+0000 7f27b608f640 1 -- 172.21.15.111:0/2724276643 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27b004f730 con 0x7f27b00a96b0 2024-03-25T19:35:31.613 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:31.613+0000 7f882064b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:35:31.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.662+0000 7f27acff9640 1 -- 172.21.15.111:0/2724276643 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27a0001220 con 0x7f27b00a96b0 2024-03-25T19:35:31.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.662+0000 7f27acff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:31.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.662+0000 7f27acff9640 1 -- 172.21.15.111:0/2724276643 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f27a0004c30 con 0x7f27b00a96b0 2024-03-25T19:35:31.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.662+0000 7f27acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:31.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.662+0000 7f27acff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:31.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.662+0000 7f27acff9640 10 monclient: dump: 2024-03-25T19:35:31.663 INFO:teuthology.orchestra.run.smithi111.stderr:epoch 1 2024-03-25T19:35:31.663 INFO:teuthology.orchestra.run.smithi111.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:31.663 INFO:teuthology.orchestra.run.smithi111.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:31.663 INFO:teuthology.orchestra.run.smithi111.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:31.663 INFO:teuthology.orchestra.run.smithi111.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:31.663 INFO:teuthology.orchestra.run.smithi111.stderr:election_strategy: 3 2024-03-25T19:35:31.663 INFO:teuthology.orchestra.run.smithi111.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:31.664 INFO:teuthology.orchestra.run.smithi111.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:31.664 INFO:teuthology.orchestra.run.smithi111.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:31.664 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:35:31.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.662+0000 7f27acff9640 1 -- 172.21.15.111:0/2724276643 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f27a0005120 con 0x7f27b00a96b0 2024-03-25T19:35:31.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.662+0000 7f27ae7fc640 4 set_mon_vals no callback set 2024-03-25T19:35:31.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.662+0000 7f27acff9640 1 -- 172.21.15.111:0/2724276643 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2751508775 0 0) 0x7f27a00068e0 con 0x7f27b00a96b0 2024-03-25T19:35:31.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.662+0000 7f27acff9640 10 client.4156.objecter ms_dispatch 0x7f27b0000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T19:35:31.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.662+0000 7f27acff9640 3 client.4156.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T19:35:31.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.662+0000 7f27acff9640 3 client.4156.objecter handle_osd_map decoding full epoch 9 2024-03-25T19:35:31.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.662+0000 7f27acff9640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-25T19:35:31.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.662+0000 7f27acff9640 1 -- 172.21.15.111:0/2724276643 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f27a0018dc0 con 0x7f27b00a96b0 2024-03-25T19:35:31.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.662+0000 7f27acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:31.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.662+0000 7f27acff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:31.665 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:31.666+0000 7f882064b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:35:31.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.735+0000 7f27b608f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f504270b-9a52-449e-8e05-e65bde3085cc", "id": 7}] 2024-03-25T19:35:31.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.735+0000 7f27b608f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f504270b-9a52-449e-8e05-e65bde3085cc", "id": 7}] 2024-03-25T19:35:31.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.735+0000 7f27b608f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:31.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.735+0000 7f27b608f640 1 -- 172.21.15.111:0/2724276643 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f504270b-9a52-449e-8e05-e65bde3085cc", "id": 7} v 0) v1 -- 0x7f27b01d78b0 con 0x7f27b00a96b0 2024-03-25T19:35:31.783 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:31.783+0000 7f882064b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:35:31.801 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.802+0000 7f27acff9640 1 -- 172.21.15.111:0/2724276643 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f504270b-9a52-449e-8e05-e65bde3085cc", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 2465380471 0 1371665406) 0x7f27a0006350 con 0x7f27b00a96b0 2024-03-25T19:35:31.801 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.802+0000 7f27acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f504270b-9a52-449e-8e05-e65bde3085cc", "id": 7}] 2024-03-25T19:35:31.801 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.802+0000 7f27acff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:31.801 INFO:teuthology.orchestra.run.smithi111.stdout:7 2024-03-25T19:35:31.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.802+0000 7f27b608f640 10 librados: watch_flush enter 2024-03-25T19:35:31.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.802+0000 7f27b608f640 10 librados: watch_flush exit 2024-03-25T19:35:31.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.802+0000 7f27b608f640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-25T19:35:31.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.802+0000 7f27b608f640 10 client.4156.objecter successfully canceled tick 2024-03-25T19:35:31.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.803+0000 7f27b608f640 10 monclient: shutdown 2024-03-25T19:35:31.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.803+0000 7f27b608f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:31.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.803+0000 7f27b608f640 1 -- 172.21.15.111:0/2724276643 >> v1:172.21.15.111:6789/0 conn(0x7f27b00a96b0 legacy=0x7f27b01d7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:31.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.803+0000 7f27b608f640 1 -- 172.21.15.111:0/2724276643 shutdown_connections 2024-03-25T19:35:31.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.803+0000 7f27b608f640 1 -- 172.21.15.111:0/2724276643 >> 172.21.15.111:0/2724276643 conn(0x7f27b00668e0 msgr2=0x7f27b00b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:31.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.803+0000 7f27b608f640 1 -- 172.21.15.111:0/2724276643 shutdown_connections 2024-03-25T19:35:31.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.803+0000 7f27b608f640 1 -- 172.21.15.111:0/2724276643 wait complete. 2024-03-25T19:35:31.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-25T19:35:31.803+0000 7f27b608f640 1 librados: shutdown 2024-03-25T19:35:31.813 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:35:31.813 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T19:35:31.813 DEBUG:teuthology.orchestra.run.smithi044:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T19:35:31.814 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T19:35:31.814+0000 7f54ee4c7040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:35:31.815 INFO:tasks.ceph.osd.0:Started 2024-03-25T19:35:31.816 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:35:31.816 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T19:35:31.816 DEBUG:teuthology.orchestra.run.smithi044:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T19:35:31.818 INFO:tasks.ceph.osd.1:Started 2024-03-25T19:35:31.818 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:35:31.818 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T19:35:31.819 DEBUG:teuthology.orchestra.run.smithi044:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T19:35:31.821 INFO:tasks.ceph.osd.2:Started 2024-03-25T19:35:31.821 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:35:31.821 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T19:35:31.821 DEBUG:teuthology.orchestra.run.smithi044:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T19:35:31.824 INFO:tasks.ceph.osd.3:Started 2024-03-25T19:35:31.824 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:35:31.824 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T19:35:31.824 DEBUG:teuthology.orchestra.run.smithi111:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T19:35:31.826 INFO:tasks.ceph.osd.4:Started 2024-03-25T19:35:31.826 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:35:31.826 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T19:35:31.826 DEBUG:teuthology.orchestra.run.smithi111:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T19:35:31.828 INFO:tasks.ceph.osd.5:Started 2024-03-25T19:35:31.828 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:35:31.828 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T19:35:31.829 DEBUG:teuthology.orchestra.run.smithi111:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T19:35:31.830 INFO:tasks.ceph.osd.6:Started 2024-03-25T19:35:31.831 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:35:31.831 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T19:35:31.831 DEBUG:teuthology.orchestra.run.smithi111:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T19:35:31.832 INFO:tasks.ceph.osd.7:Started 2024-03-25T19:35:31.833 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:31.904 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T19:35:31.904+0000 7f54ee4c7040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:35:31.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:31.996+0000 7f254cdaf640 10 monclient: get_monmap_and_config 2024-03-25T19:35:31.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:31.996+0000 7f254cdaf640 10 monclient: build_initial_monmap 2024-03-25T19:35:31.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:31.997+0000 7f254cdaf640 10 monclient: monmap: 2024-03-25T19:35:31.997 INFO:teuthology.orchestra.run.smithi044.stderr:epoch 0 2024-03-25T19:35:31.997 INFO:teuthology.orchestra.run.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:31.997 INFO:teuthology.orchestra.run.smithi044.stderr:last_changed 2024-03-25T19:35:31.998133+0000 2024-03-25T19:35:31.997 INFO:teuthology.orchestra.run.smithi044.stderr:created 2024-03-25T19:35:31.998133+0000 2024-03-25T19:35:31.997 INFO:teuthology.orchestra.run.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:31.997 INFO:teuthology.orchestra.run.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:31.997 INFO:teuthology.orchestra.run.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:31.998 INFO:teuthology.orchestra.run.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:35:31.998 INFO:teuthology.orchestra.run.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:35:31.998 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T19:35:32.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:31.997+0000 7f254cdaf640 1 Processor -- start 2024-03-25T19:35:32.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:31.997+0000 7f254cdaf640 1 -- start start 2024-03-25T19:35:32.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:31.997+0000 7f254cdaf640 10 monclient: init 2024-03-25T19:35:32.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:31.998+0000 7f254cdaf640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:32.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:31.998+0000 7f254cdaf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:35:32.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:31.998+0000 7f254cdaf640 10 monclient(hunting): picked mon.noname-b con 0x7f25480aeeb0 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:32.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:31.998+0000 7f254cdaf640 10 monclient(hunting): picked mon.noname-a con 0x7f25480ac9d0 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:32.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:31.998+0000 7f254cdaf640 10 monclient(hunting): picked mon.noname-c con 0x7f254813efc0 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:32.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:31.998+0000 7f254cdaf640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25480b0340 con 0x7f25480ac9d0 2024-03-25T19:35:32.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:31.998+0000 7f254cdaf640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25480729c0 con 0x7f254813efc0 2024-03-25T19:35:32.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:31.998+0000 7f254cdaf640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25480abbc0 con 0x7f25480aeeb0 2024-03-25T19:35:32.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:31.998+0000 7f254cdaf640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:32.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:31.998+0000 7f254cdaf640 10 monclient(hunting): authenticate will time out at 5396.169825s 2024-03-25T19:35:32.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.001+0000 7f2546d9d640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f254813efc0 0x7f254814a0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:37766/0 (socket says 172.21.15.44:37766) 2024-03-25T19:35:32.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.001+0000 7f2546d9d640 1 -- 172.21.15.44:0/345657601 learned_addr learned my addr 172.21.15.44:0/345657601 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:35:32.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.004+0000 7f254559a640 1 -- 172.21.15.44:0/345657601 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2511468398 0 0) 0x7f25480b0340 con 0x7f25480ac9d0 2024-03-25T19:35:32.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.004+0000 7f254559a640 10 monclient(hunting): _negotiate 2024-03-25T19:35:32.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.004+0000 7f254559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:32.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.004+0000 7f254559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:32.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.004+0000 7f254559a640 10 monclient(hunting): my global_id is 4164 2024-03-25T19:35:32.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.004+0000 7f254559a640 1 -- 172.21.15.44:0/345657601 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2530003520 con 0x7f25480ac9d0 2024-03-25T19:35:32.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.007+0000 7f254559a640 1 -- 172.21.15.44:0/345657601 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 110444087 0 0) 0x7f2530003520 con 0x7f25480ac9d0 2024-03-25T19:35:32.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.008+0000 7f254559a640 1 -- 172.21.15.44:0/345657601 >> v1:172.21.15.44:6790/0 conn(0x7f25480aeeb0 legacy=0x7f25480ac2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:35:32.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.008+0000 7f254559a640 1 -- 172.21.15.44:0/345657601 >> v1:172.21.15.111:6789/0 conn(0x7f254813efc0 legacy=0x7f254814a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:35:32.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.008+0000 7f254559a640 10 monclient: _finish_hunting 0 2024-03-25T19:35:32.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.008+0000 7f254559a640 1 monclient: found mon.noname-a 2024-03-25T19:35:32.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.008+0000 7f254559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:32.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.008+0000 7f254559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:35:32.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.008+0000 7f254559a640 1 -- 172.21.15.44:0/345657601 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25480afce0 con 0x7f25480ac9d0 2024-03-25T19:35:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.008+0000 7f254559a640 10 monclient: _finish_auth 0 2024-03-25T19:35:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.008+0000 7f254559a640 10 monclient: _check_auth_tickets 2024-03-25T19:35:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.008+0000 7f254559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.008+0000 7f254cdaf640 5 monclient: authenticate success, global_id 4164 2024-03-25T19:35:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.008+0000 7f254cdaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.008+0000 7f254559a640 1 -- 172.21.15.44:0/345657601 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f2538001690 con 0x7f25480ac9d0 2024-03-25T19:35:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.008+0000 7f254559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.008+0000 7f254559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.008+0000 7f254559a640 10 monclient: dump: 2024-03-25T19:35:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:epoch 1 2024-03-25T19:35:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:32.010 INFO:teuthology.orchestra.run.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:32.010 INFO:teuthology.orchestra.run.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:32.010 INFO:teuthology.orchestra.run.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:32.010 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T19:35:32.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.008+0000 7f254cdaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:32.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.011+0000 7f254559a640 1 -- 172.21.15.44:0/345657601 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25380018f0 con 0x7f25480ac9d0 2024-03-25T19:35:32.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.011+0000 7f254559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:32.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.011+0000 7f254cdaf640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:32.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.011+0000 7f254cdaf640 4 set_mon_vals no callback set 2024-03-25T19:35:32.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.012+0000 7f254cdaf640 10 monclient: shutdown 2024-03-25T19:35:32.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.012+0000 7f254cdaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:32.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.012+0000 7f254cdaf640 1 -- 172.21.15.44:0/345657601 >> v1:172.21.15.44:6789/0 conn(0x7f25480ac9d0 legacy=0x7f254813e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:32.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.013+0000 7f254cdaf640 1 -- 172.21.15.44:0/345657601 shutdown_connections 2024-03-25T19:35:32.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.013+0000 7f254cdaf640 1 -- 172.21.15.44:0/345657601 >> 172.21.15.44:0/345657601 conn(0x7f254806b560 msgr2=0x7f25480b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:32.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.013+0000 7f254cdaf640 1 -- 172.21.15.44:0/345657601 shutdown_connections 2024-03-25T19:35:32.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.013+0000 7f254cdaf640 1 -- 172.21.15.44:0/345657601 wait complete. 2024-03-25T19:35:32.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.014+0000 7f254cdaf640 10 monclient: build_initial_monmap 2024-03-25T19:35:32.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.014+0000 7f254cdaf640 10 monclient: monmap: 2024-03-25T19:35:32.015 INFO:teuthology.orchestra.run.smithi044.stderr:epoch 0 2024-03-25T19:35:32.015 INFO:teuthology.orchestra.run.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:32.015 INFO:teuthology.orchestra.run.smithi044.stderr:last_changed 0.000000 2024-03-25T19:35:32.015 INFO:teuthology.orchestra.run.smithi044.stderr:created 0.000000 2024-03-25T19:35:32.015 INFO:teuthology.orchestra.run.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:32.015 INFO:teuthology.orchestra.run.smithi044.stderr:election_strategy: 1 2024-03-25T19:35:32.015 INFO:teuthology.orchestra.run.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:32.016 INFO:teuthology.orchestra.run.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:35:32.016 INFO:teuthology.orchestra.run.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:35:32.016 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T19:35:32.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.015+0000 7f254cdaf640 1 librados: starting msgr at 2024-03-25T19:35:32.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.015+0000 7f254cdaf640 1 librados: starting objecter 2024-03-25T19:35:32.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.015+0000 7f254cdaf640 1 Processor -- start 2024-03-25T19:35:32.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.015+0000 7f254cdaf640 1 -- start start 2024-03-25T19:35:32.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.015+0000 7f254cdaf640 1 librados: setting wanted keys 2024-03-25T19:35:32.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.015+0000 7f254cdaf640 1 librados: calling monclient init 2024-03-25T19:35:32.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.015+0000 7f254cdaf640 10 monclient: init 2024-03-25T19:35:32.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.016+0000 7f254cdaf640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:32.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.016+0000 7f254cdaf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:35:32.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.016+0000 7f254cdaf640 10 monclient(hunting): picked mon.noname-a con 0x7f25480ac9d0 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:32.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.016+0000 7f254cdaf640 10 monclient(hunting): picked mon.noname-c con 0x7f25480aeeb0 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:32.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.016+0000 7f254cdaf640 10 monclient(hunting): picked mon.noname-b con 0x7f254813efc0 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:32.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.016+0000 7f254cdaf640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25480725a0 con 0x7f25480ac9d0 2024-03-25T19:35:32.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.016+0000 7f254cdaf640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25481e4d40 con 0x7f254813efc0 2024-03-25T19:35:32.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.016+0000 7f254cdaf640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25480716f0 con 0x7f25480aeeb0 2024-03-25T19:35:32.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.016+0000 7f254cdaf640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:32.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.016+0000 7f254cdaf640 10 monclient(hunting): authenticate will time out at 5396.187193s 2024-03-25T19:35:32.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.017+0000 7f254659c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f25480ac9d0 0x7f2548071e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:44596/0 (socket says 172.21.15.44:44596) 2024-03-25T19:35:32.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.017+0000 7f254659c640 1 -- 172.21.15.44:0/3090519547 learned_addr learned my addr 172.21.15.44:0/3090519547 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:35:32.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.017+0000 7f2536ffd640 10 client.?.objecter ms_handle_connect 0x7f25480ac9d0 2024-03-25T19:35:32.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.017+0000 7f2536ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:32.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.017+0000 7f2536ffd640 10 client.?.objecter ms_handle_connect 0x7f25480aeeb0 2024-03-25T19:35:32.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.017+0000 7f2536ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:32.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.019+0000 7f2536ffd640 1 -- 172.21.15.44:0/3090519547 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201652438 0 0) 0x7f25480725a0 con 0x7f25480ac9d0 2024-03-25T19:35:32.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.019+0000 7f2536ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:32.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.019+0000 7f2536ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:32.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.019+0000 7f2536ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:32.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.019+0000 7f2536ffd640 10 monclient(hunting): my global_id is 4167 2024-03-25T19:35:32.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.019+0000 7f2536ffd640 1 -- 172.21.15.44:0/3090519547 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2528004eb0 con 0x7f25480ac9d0 2024-03-25T19:35:32.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.023+0000 7f2536ffd640 10 client.?.objecter ms_handle_connect 0x7f254813efc0 2024-03-25T19:35:32.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.023+0000 7f2536ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:32.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.026+0000 7f2536ffd640 1 -- 172.21.15.44:0/3090519547 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 969875143 0 0) 0x7f2528004eb0 con 0x7f25480ac9d0 2024-03-25T19:35:32.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.026+0000 7f2536ffd640 1 -- 172.21.15.44:0/3090519547 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25480725a0 con 0x7f25480ac9d0 2024-03-25T19:35:32.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.026+0000 7f2536ffd640 1 -- 172.21.15.44:0/3090519547 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f253c001960 con 0x7f25480ac9d0 2024-03-25T19:35:32.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.026+0000 7f2536ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:32.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.026+0000 7f2536ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:32.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.026+0000 7f2536ffd640 10 monclient(hunting): dump: 2024-03-25T19:35:32.026 INFO:teuthology.orchestra.run.smithi044.stderr:epoch 1 2024-03-25T19:35:32.026 INFO:teuthology.orchestra.run.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:32.027 INFO:teuthology.orchestra.run.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:32.027 INFO:teuthology.orchestra.run.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:32.027 INFO:teuthology.orchestra.run.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:32.027 INFO:teuthology.orchestra.run.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:32.027 INFO:teuthology.orchestra.run.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:32.027 INFO:teuthology.orchestra.run.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:32.027 INFO:teuthology.orchestra.run.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:32.027 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T19:35:32.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.026+0000 7f2536ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:32.118 INFO:tasks.ceph.mgr.y.smithi044.stderr:2024-03-25T19:35:32.118+0000 7f54ee4c7040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:35:32.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.211+0000 7f2536ffd640 1 -- 172.21.15.44:0/3090519547 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1362111108 0 0) 0x7f25481e4d40 con 0x7f254813efc0 2024-03-25T19:35:32.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.211+0000 7f2536ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:32.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.211+0000 7f2536ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:32.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.211+0000 7f2536ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:32.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.211+0000 7f2536ffd640 10 monclient(hunting): my global_id is 4159 2024-03-25T19:35:32.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.211+0000 7f2536ffd640 1 -- 172.21.15.44:0/3090519547 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2528004eb0 con 0x7f254813efc0 2024-03-25T19:35:32.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.217+0000 7f2536ffd640 1 -- 172.21.15.44:0/3090519547 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 254331248 0 0) 0x7f25480725a0 con 0x7f25480ac9d0 2024-03-25T19:35:32.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.217+0000 7f2536ffd640 1 -- 172.21.15.44:0/3090519547 >> v1:172.21.15.44:6790/0 conn(0x7f25480aeeb0 legacy=0x7f254806f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:32.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.217+0000 7f2536ffd640 1 -- 172.21.15.44:0/3090519547 >> v1:172.21.15.111:6789/0 conn(0x7f254813efc0 legacy=0x7f25481e1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:32.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.217+0000 7f2536ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:35:32.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.217+0000 7f2536ffd640 1 monclient: found mon.a 2024-03-25T19:35:32.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.217+0000 7f2536ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:32.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.217+0000 7f2536ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:32.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.217+0000 7f2536ffd640 1 -- 172.21.15.44:0/3090519547 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254806e580 con 0x7f25480ac9d0 2024-03-25T19:35:32.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.217+0000 7f2536ffd640 10 monclient: _finish_auth 0 2024-03-25T19:35:32.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.217+0000 7f2536ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:35:32.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.217+0000 7f2536ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:32.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.217+0000 7f254cdaf640 5 monclient: authenticate success, global_id 4167 2024-03-25T19:35:32.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.217+0000 7f254cdaf640 10 monclient: _renew_subs 2024-03-25T19:35:32.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.217+0000 7f254cdaf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:32.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.217+0000 7f254cdaf640 1 -- 172.21.15.44:0/3090519547 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25480719f0 con 0x7f25480ac9d0 2024-03-25T19:35:32.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.217+0000 7f254cdaf640 10 client.4167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:32.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.217+0000 7f254cdaf640 10 monclient: _renew_subs 2024-03-25T19:35:32.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.217+0000 7f254cdaf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:32.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.217+0000 7f254cdaf640 1 -- 172.21.15.44:0/3090519547 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25481e19f0 con 0x7f25480ac9d0 2024-03-25T19:35:32.218 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.217+0000 7f254cdaf640 1 librados: init done 2024-03-25T19:35:32.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.218+0000 7f254cdaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:32.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.218+0000 7f254cdaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:32.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.218+0000 7f254cdaf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:32.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.218+0000 7f254cdaf640 1 -- 172.21.15.44:0/3090519547 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25480ab0f0 con 0x7f25480ac9d0 2024-03-25T19:35:32.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.240+0000 7f2536ffd640 1 -- 172.21.15.44:0/3090519547 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f253c0035f0 con 0x7f25480ac9d0 2024-03-25T19:35:32.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.240+0000 7f2536ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:32.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.240+0000 7f2537fff640 4 set_mon_vals no callback set 2024-03-25T19:35:32.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.263+0000 7f2536ffd640 1 -- 172.21.15.44:0/3090519547 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f253c003980 con 0x7f25480ac9d0 2024-03-25T19:35:32.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.263+0000 7f2536ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:32.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.263+0000 7f2536ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:32.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.263+0000 7f2536ffd640 10 monclient: dump: 2024-03-25T19:35:32.263 INFO:teuthology.orchestra.run.smithi044.stderr:epoch 1 2024-03-25T19:35:32.263 INFO:teuthology.orchestra.run.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:32.263 INFO:teuthology.orchestra.run.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:32.263 INFO:teuthology.orchestra.run.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:32.263 INFO:teuthology.orchestra.run.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:32.263 INFO:teuthology.orchestra.run.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:32.264 INFO:teuthology.orchestra.run.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:32.264 INFO:teuthology.orchestra.run.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:32.264 INFO:teuthology.orchestra.run.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:32.264 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T19:35:32.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.264+0000 7f2536ffd640 1 -- 172.21.15.44:0/3090519547 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 1369965474 0 0) 0x7f253c003b60 con 0x7f25480ac9d0 2024-03-25T19:35:32.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.295+0000 7f2536ffd640 1 -- 172.21.15.44:0/3090519547 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f253c0156e0 con 0x7f25480ac9d0 2024-03-25T19:35:32.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.295+0000 7f2536ffd640 10 client.4167.objecter ms_dispatch 0x7f2548000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:32.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.295+0000 7f2536ffd640 3 client.4167.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:32.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.295+0000 7f2536ffd640 3 client.4167.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:32.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.295+0000 7f2536ffd640 20 client.4167.objecter dump_active .. 0 homeless 2024-03-25T19:35:32.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.440+0000 7f2536ffd640 1 -- 172.21.15.44:0/3090519547 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f253c0149a0 con 0x7f25480ac9d0 2024-03-25T19:35:32.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.441+0000 7f2536ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:32.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.441+0000 7f2536ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:32.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.524+0000 7f254cdaf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:32.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.524+0000 7f254cdaf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:32.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.524+0000 7f254cdaf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:32.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.524+0000 7f254cdaf640 1 -- 172.21.15.44:0/3090519547 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2548071b80 con 0x7f25480ac9d0 2024-03-25T19:35:32.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.571+0000 7f2536ffd640 1 -- 172.21.15.44:0/3090519547 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f253c014db0 con 0x7f25480ac9d0 2024-03-25T19:35:32.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.571+0000 7f2536ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:32.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.571+0000 7f2536ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:32.571 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-03-25T19:35:32.571 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:32.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.572+0000 7f254cdaf640 10 librados: watch_flush enter 2024-03-25T19:35:32.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.572+0000 7f254cdaf640 10 librados: watch_flush exit 2024-03-25T19:35:32.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.572+0000 7f254cdaf640 20 client.4167.objecter shutdown clearing up homeless session... 2024-03-25T19:35:32.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.572+0000 7f254cdaf640 10 client.4167.objecter successfully canceled tick 2024-03-25T19:35:32.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.572+0000 7f254cdaf640 10 monclient: shutdown 2024-03-25T19:35:32.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.572+0000 7f254cdaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:32.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.572+0000 7f254cdaf640 1 -- 172.21.15.44:0/3090519547 >> v1:172.21.15.44:6789/0 conn(0x7f25480ac9d0 legacy=0x7f2548071e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:32.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.572+0000 7f254cdaf640 1 -- 172.21.15.44:0/3090519547 shutdown_connections 2024-03-25T19:35:32.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.572+0000 7f254cdaf640 1 -- 172.21.15.44:0/3090519547 >> 172.21.15.44:0/3090519547 conn(0x7f254806b560 msgr2=0x7f2548074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:32.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.573+0000 7f254cdaf640 1 -- 172.21.15.44:0/3090519547 shutdown_connections 2024-03-25T19:35:32.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.573+0000 7f254cdaf640 1 -- 172.21.15.44:0/3090519547 wait complete. 2024-03-25T19:35:32.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-03-25T19:35:32.573+0000 7f254cdaf640 1 librados: shutdown 2024-03-25T19:35:32.584 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T19:35:32.584 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T19:35:33.389 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:33.536 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.536+0000 7f669d7a4640 10 monclient: get_monmap_and_config 2024-03-25T19:35:33.536 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.536+0000 7f669d7a4640 10 monclient: build_initial_monmap 2024-03-25T19:35:33.536 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.536+0000 7f669d7a4640 10 monclient: monmap: 2024-03-25T19:35:33.536 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:35:33.536 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:33.536 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:35:33.537702+0000 2024-03-25T19:35:33.537 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:35:33.537702+0000 2024-03-25T19:35:33.537 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:33.537 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:33.537 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:33.537 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:35:33.537 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:35:33.537 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:33.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.537+0000 7f669d7a4640 1 Processor -- start 2024-03-25T19:35:33.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.537+0000 7f669d7a4640 1 -- start start 2024-03-25T19:35:33.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.537+0000 7f669d7a4640 10 monclient: init 2024-03-25T19:35:33.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.538+0000 7f669d7a4640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:33.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.538+0000 7f669d7a4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:33.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.538+0000 7f669d7a4640 10 monclient(hunting): picked mon.noname-c con 0x7f66980a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:33.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.538+0000 7f669d7a4640 10 monclient(hunting): picked mon.noname-a con 0x7f669813f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:33.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.538+0000 7f669d7a4640 10 monclient(hunting): picked mon.noname-b con 0x7f66981430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:33.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.538+0000 7f669d7a4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6698142520 con 0x7f669813f710 2024-03-25T19:35:33.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.538+0000 7f669d7a4640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66980b28a0 con 0x7f66980a98f0 2024-03-25T19:35:33.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.538+0000 7f669d7a4640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66980b2a30 con 0x7f66981430c0 2024-03-25T19:35:33.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.538+0000 7f669d7a4640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:33.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.538+0000 7f669d7a4640 10 monclient(hunting): authenticate will time out at 5397.709537s 2024-03-25T19:35:33.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.539+0000 7f6696ffd640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f66980a98f0 0x7f66980a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:37778/0 (socket says 172.21.15.44:37778) 2024-03-25T19:35:33.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.539+0000 7f6696ffd640 1 -- 172.21.15.44:0/3876296557 learned_addr learned my addr 172.21.15.44:0/3876296557 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:35:33.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.768+0000 7f6695ffb640 1 -- 172.21.15.44:0/3876296557 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1446473892 0 0) 0x7f6698142520 con 0x7f669813f710 2024-03-25T19:35:33.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.768+0000 7f6695ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:33.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.768+0000 7f6695ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:33.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.768+0000 7f6695ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:33.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.768+0000 7f6695ffb640 10 monclient(hunting): my global_id is 4188 2024-03-25T19:35:33.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.768+0000 7f6695ffb640 1 -- 172.21.15.44:0/3876296557 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6680003520 con 0x7f669813f710 2024-03-25T19:35:33.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.779+0000 7f6695ffb640 1 -- 172.21.15.44:0/3876296557 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1019335615 0 0) 0x7f6680003520 con 0x7f669813f710 2024-03-25T19:35:33.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.779+0000 7f6695ffb640 1 -- 172.21.15.44:0/3876296557 >> v1:172.21.15.44:6790/0 conn(0x7f66981430c0 legacy=0x7f669814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:33.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.779+0000 7f6695ffb640 1 -- 172.21.15.44:0/3876296557 >> v1:172.21.15.111:6789/0 conn(0x7f66980a98f0 legacy=0x7f66980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:33.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.779+0000 7f6695ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:35:33.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.779+0000 7f6695ffb640 1 monclient: found mon.noname-a 2024-03-25T19:35:33.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.779+0000 7f6695ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:33.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.779+0000 7f6695ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:35:33.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.779+0000 7f6695ffb640 1 -- 172.21.15.44:0/3876296557 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66980b44c0 con 0x7f669813f710 2024-03-25T19:35:33.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.779+0000 7f6695ffb640 10 monclient: _finish_auth 0 2024-03-25T19:35:33.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.779+0000 7f6695ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:35:33.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.779+0000 7f6695ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:33.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.779+0000 7f6695ffb640 1 -- 172.21.15.44:0/3876296557 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f668c001670 con 0x7f669813f710 2024-03-25T19:35:33.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.779+0000 7f6695ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:33.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.779+0000 7f6695ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:33.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.779+0000 7f6695ffb640 10 monclient: dump: 2024-03-25T19:35:33.780 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:35:33.780 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:33.780 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:33.780 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:33.780 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:33.780 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:33.780 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:33.780 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:33.781 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:33.781 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:33.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.779+0000 7f669d7a4640 5 monclient: authenticate success, global_id 4188 2024-03-25T19:35:33.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.779+0000 7f669d7a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:33.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.782+0000 7f6695ffb640 1 -- 172.21.15.44:0/3876296557 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f668c0017d0 con 0x7f669813f710 2024-03-25T19:35:33.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.782+0000 7f6695ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:33.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.782+0000 7f6695ffb640 1 -- 172.21.15.44:0/3876296557 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f668c001c20 con 0x7f669813f710 2024-03-25T19:35:33.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.782+0000 7f6695ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:33.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.782+0000 7f6695ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:33.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.782+0000 7f6695ffb640 10 monclient: dump: 2024-03-25T19:35:33.783 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:35:33.783 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:33.783 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:33.783 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:33.783 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:33.783 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:33.783 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:33.783 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:33.783 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:33.783 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:33.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.782+0000 7f669d7a4640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:33.784 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.782+0000 7f669d7a4640 4 set_mon_vals no callback set 2024-03-25T19:35:33.784 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.783+0000 7f669d7a4640 10 monclient: shutdown 2024-03-25T19:35:33.785 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.783+0000 7f669d7a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:33.785 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.783+0000 7f669d7a4640 1 -- 172.21.15.44:0/3876296557 >> v1:172.21.15.44:6789/0 conn(0x7f669813f710 legacy=0x7f66981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:33.785 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.783+0000 7f669d7a4640 1 -- 172.21.15.44:0/3876296557 shutdown_connections 2024-03-25T19:35:33.785 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.783+0000 7f669d7a4640 1 -- 172.21.15.44:0/3876296557 >> 172.21.15.44:0/3876296557 conn(0x7f66980b7ea0 msgr2=0x7f66980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:33.785 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.783+0000 7f669d7a4640 1 -- 172.21.15.44:0/3876296557 shutdown_connections 2024-03-25T19:35:33.785 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.783+0000 7f669d7a4640 1 -- 172.21.15.44:0/3876296557 wait complete. 2024-03-25T19:35:33.785 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.784+0000 7f669d7a4640 10 monclient: build_initial_monmap 2024-03-25T19:35:33.785 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.784+0000 7f669d7a4640 10 monclient: monmap: 2024-03-25T19:35:33.785 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:35:33.785 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:33.785 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:35:33.785 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:35:33.785 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:33.785 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:35:33.786 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:33.786 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:35:33.786 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:35:33.786 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:33.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.785+0000 7f669d7a4640 1 librados: starting msgr at 2024-03-25T19:35:33.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.785+0000 7f669d7a4640 1 librados: starting objecter 2024-03-25T19:35:33.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.785+0000 7f669d7a4640 1 Processor -- start 2024-03-25T19:35:33.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.786+0000 7f669d7a4640 1 -- start start 2024-03-25T19:35:33.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.786+0000 7f669d7a4640 1 librados: setting wanted keys 2024-03-25T19:35:33.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.786+0000 7f669d7a4640 1 librados: calling monclient init 2024-03-25T19:35:33.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.786+0000 7f669d7a4640 10 monclient: init 2024-03-25T19:35:33.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.787+0000 7f669d7a4640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:33.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.787+0000 7f669d7a4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:35:33.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.787+0000 7f669d7a4640 10 monclient(hunting): picked mon.noname-c con 0x7f66980a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:33.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.787+0000 7f669d7a4640 10 monclient(hunting): picked mon.noname-a con 0x7f669813f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:33.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.787+0000 7f669d7a4640 10 monclient(hunting): picked mon.noname-b con 0x7f66981430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:33.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.787+0000 7f669d7a4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6698135d40 con 0x7f669813f710 2024-03-25T19:35:33.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.787+0000 7f669d7a4640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66981358b0 con 0x7f66981430c0 2024-03-25T19:35:33.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.787+0000 7f669d7a4640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6698135a40 con 0x7f66980a98f0 2024-03-25T19:35:33.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.787+0000 7f669d7a4640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:33.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.787+0000 7f669d7a4640 10 monclient(hunting): authenticate will time out at 5397.958740s 2024-03-25T19:35:33.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.788+0000 7f66967fc640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f669813f710 0x7f66981daae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:44624/0 (socket says 172.21.15.44:44624) 2024-03-25T19:35:33.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.788+0000 7f66967fc640 1 -- 172.21.15.44:0/3979419696 learned_addr learned my addr 172.21.15.44:0/3979419696 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:35:33.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.789+0000 7f667f7fe640 10 client.?.objecter ms_handle_connect 0x7f669813f710 2024-03-25T19:35:33.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.789+0000 7f667f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:33.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.792+0000 7f667f7fe640 1 -- 172.21.15.44:0/3979419696 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206603244 0 0) 0x7f6698135d40 con 0x7f669813f710 2024-03-25T19:35:33.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.792+0000 7f667f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:33.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.792+0000 7f667f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:33.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.792+0000 7f667f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:33.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.792+0000 7f667f7fe640 10 monclient(hunting): my global_id is 4191 2024-03-25T19:35:33.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.792+0000 7f667f7fe640 1 -- 172.21.15.44:0/3979419696 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6684004eb0 con 0x7f669813f710 2024-03-25T19:35:33.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.800+0000 7f667f7fe640 1 -- 172.21.15.44:0/3979419696 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 332802463 0 0) 0x7f6684004eb0 con 0x7f669813f710 2024-03-25T19:35:33.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.800+0000 7f667f7fe640 1 -- 172.21.15.44:0/3979419696 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6684005300 con 0x7f669813f710 2024-03-25T19:35:33.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.800+0000 7f667f7fe640 1 -- 172.21.15.44:0/3979419696 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f668c001670 con 0x7f669813f710 2024-03-25T19:35:33.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.800+0000 7f667f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:33.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.800+0000 7f667f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:33.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.800+0000 7f667f7fe640 10 monclient(hunting): dump: 2024-03-25T19:35:33.801 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:35:33.801 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:33.801 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:33.801 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:33.801 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:33.801 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:33.801 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:33.801 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:33.801 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:33.801 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:33.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.800+0000 7f667f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:33.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.806+0000 7f667f7fe640 1 -- 172.21.15.44:0/3979419696 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3405959193 0 0) 0x7f6684005300 con 0x7f669813f710 2024-03-25T19:35:33.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.806+0000 7f667f7fe640 1 -- 172.21.15.44:0/3979419696 >> v1:172.21.15.44:6790/0 conn(0x7f66980a98f0 legacy=0x7f66981351a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:33.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.806+0000 7f667f7fe640 1 -- 172.21.15.44:0/3979419696 >> v1:172.21.15.111:6789/0 conn(0x7f66981430c0 legacy=0x7f66981db1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:33.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.806+0000 7f667f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:35:33.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.806+0000 7f667f7fe640 1 monclient: found mon.a 2024-03-25T19:35:33.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.806+0000 7f667f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:33.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.806+0000 7f667f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:33.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.806+0000 7f667f7fe640 1 -- 172.21.15.44:0/3979419696 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6698141bf0 con 0x7f669813f710 2024-03-25T19:35:33.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.806+0000 7f667f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:35:33.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.806+0000 7f667f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:35:33.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.806+0000 7f667f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:33.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.806+0000 7f669d7a4640 5 monclient: authenticate success, global_id 4191 2024-03-25T19:35:33.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.806+0000 7f669d7a4640 10 monclient: _renew_subs 2024-03-25T19:35:33.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.806+0000 7f669d7a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:33.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.806+0000 7f669d7a4640 1 -- 172.21.15.44:0/3979419696 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6698141ea0 con 0x7f669813f710 2024-03-25T19:35:33.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.806+0000 7f669d7a4640 10 client.4191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:33.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.806+0000 7f669d7a4640 10 monclient: _renew_subs 2024-03-25T19:35:33.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.806+0000 7f669d7a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:33.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.806+0000 7f669d7a4640 1 -- 172.21.15.44:0/3979419696 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f66981e3150 con 0x7f669813f710 2024-03-25T19:35:33.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.806+0000 7f669d7a4640 1 librados: init done 2024-03-25T19:35:33.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.807+0000 7f669d7a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:33.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.807+0000 7f669d7a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:33.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.807+0000 7f669d7a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:33.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.807+0000 7f669d7a4640 1 -- 172.21.15.44:0/3979419696 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66981e3150 con 0x7f669813f710 2024-03-25T19:35:33.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.820+0000 7f667f7fe640 1 -- 172.21.15.44:0/3979419696 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f668c003a40 con 0x7f669813f710 2024-03-25T19:35:33.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.820+0000 7f667f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:33.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.820+0000 7f667f7fe640 1 -- 172.21.15.44:0/3979419696 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f668c001670 con 0x7f669813f710 2024-03-25T19:35:33.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.820+0000 7f667f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:33.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.820+0000 7f667f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:33.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.820+0000 7f667f7fe640 10 monclient: dump: 2024-03-25T19:35:33.820 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:35:33.820 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:33.820 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:33.821 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:33.821 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:33.821 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:33.821 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:33.821 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:33.821 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:33.821 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:33.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.820+0000 7f66957fa640 4 set_mon_vals no callback set 2024-03-25T19:35:33.838 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.838+0000 7f667f7fe640 1 -- 172.21.15.44:0/3979419696 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 864801898 0 0) 0x7f668c003c70 con 0x7f669813f710 2024-03-25T19:35:33.838 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.839+0000 7f6696ffd640 10 monclient: get_auth_request con 0x7f668406e010 auth_method 0 2024-03-25T19:35:33.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.839+0000 7f667f7fe640 10 client.4191.objecter ms_handle_connect 0x7f668406e010 2024-03-25T19:35:33.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.872+0000 7f667f7fe640 1 -- 172.21.15.44:0/3979419696 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f668c054680 con 0x7f669813f710 2024-03-25T19:35:33.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.872+0000 7f667f7fe640 10 client.4191.objecter ms_dispatch 0x7f6698000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:33.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.872+0000 7f667f7fe640 3 client.4191.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:33.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.872+0000 7f667f7fe640 3 client.4191.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:33.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:33.872+0000 7f667f7fe640 20 client.4191.objecter dump_active .. 0 homeless 2024-03-25T19:35:34.055 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:34.056+0000 7f87c1382640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T19:35:34.124 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:35:34.124+0000 7f87c1382640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T19:35:34.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:34.252+0000 7f667f7fe640 1 -- 172.21.15.44:0/3979419696 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f668c050920 con 0x7f669813f710 2024-03-25T19:35:34.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:34.252+0000 7f667f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:34.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:34.252+0000 7f667f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:34.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:34.596+0000 7f669d7a4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:34.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:34.596+0000 7f669d7a4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:34.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:34.596+0000 7f669d7a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:35:34.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:34.596+0000 7f669d7a4640 1 -- 172.21.15.44:0/3979419696 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66981e3150 con 0x7f669813f710 2024-03-25T19:35:34.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:34.624+0000 7f667f7fe640 1 -- 172.21.15.44:0/3979419696 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f668c050920 con 0x7f669813f710 2024-03-25T19:35:34.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:34.624+0000 7f667f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:34.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:34.624+0000 7f667f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:34.625 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:35:34.625 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:34.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:34.627+0000 7f669d7a4640 10 librados: watch_flush enter 2024-03-25T19:35:34.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:34.627+0000 7f669d7a4640 10 librados: watch_flush exit 2024-03-25T19:35:34.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:34.627+0000 7f669d7a4640 20 client.4191.objecter shutdown clearing up homeless session... 2024-03-25T19:35:34.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:34.627+0000 7f669d7a4640 10 client.4191.objecter successfully canceled tick 2024-03-25T19:35:34.628 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:34.627+0000 7f669d7a4640 1 -- 172.21.15.44:0/3979419696 >> v1:172.21.15.111:6800/3883215793 conn(0x7f668406e010 legacy=0x7f66840704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:34.628 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:34.627+0000 7f669d7a4640 10 monclient: shutdown 2024-03-25T19:35:34.628 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:34.627+0000 7f669d7a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:34.628 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:34.627+0000 7f669d7a4640 1 -- 172.21.15.44:0/3979419696 >> v1:172.21.15.44:6789/0 conn(0x7f669813f710 legacy=0x7f66981daae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:34.628 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:34.627+0000 7f669d7a4640 1 -- 172.21.15.44:0/3979419696 shutdown_connections 2024-03-25T19:35:34.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:34.627+0000 7f669d7a4640 1 -- 172.21.15.44:0/3979419696 >> 172.21.15.44:0/3979419696 conn(0x7f66980b7ea0 msgr2=0x7f669813d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:34.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:34.627+0000 7f669d7a4640 1 -- 172.21.15.44:0/3979419696 shutdown_connections 2024-03-25T19:35:34.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:34.627+0000 7f669d7a4640 1 -- 172.21.15.44:0/3979419696 wait complete. 2024-03-25T19:35:34.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:34.627+0000 7f669d7a4640 1 librados: shutdown 2024-03-25T19:35:34.642 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:38.658 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:06.764 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:38.680 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:06.787 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:38.720 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:06.825 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:38.729 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:06.839 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:38.741 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:06.848 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:38.759 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:06.870 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:38.791 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:06.901 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:38.822 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:06.933 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:39.266 INFO:tasks.ceph.osd.7.smithi111.stderr:2024-03-25T19:35:39.261+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:35:39.296 INFO:tasks.ceph.osd.6.smithi111.stderr:2024-03-25T19:35:39.290+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:35:39.346 INFO:tasks.ceph.osd.5.smithi111.stderr:2024-03-25T19:35:39.341+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:35:39.379 INFO:tasks.ceph.osd.4.smithi111.stderr:2024-03-25T19:35:39.373+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:35:39.399 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:07.490 35567== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:39.431 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:07.517 35568== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:39.447 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:07.548 35565== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:39.461 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:07.552 35567== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:39.493 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:07.579 35568== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:39.508 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:07.609 35565== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:39.583 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:07.676 35566== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:39.643 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:07.736 35566== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:39.723 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:07.828 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:39.814 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:07.924 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:39.830 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:07.937 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:39.904 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:08.015 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:40.005 INFO:tasks.ceph.osd.1.smithi044.stderr:2024-03-25T19:35:40.000+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:35:40.062 INFO:tasks.ceph.osd.3.smithi044.stderr:2024-03-25T19:35:40.055+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:35:40.079 INFO:tasks.ceph.osd.2.smithi044.stderr:2024-03-25T19:35:40.074+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:35:40.086 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:08.192 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:40.180 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:08.290 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:40.202 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:08.309 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:40.241 INFO:tasks.ceph.osd.0.smithi044.stderr:2024-03-25T19:35:40.235+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:35:40.269 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:08.381 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:40.399 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:08.505 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:40.472 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:08.563 35567== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:40.501 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:08.611 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:40.543 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:08.644 35565== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:40.557 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:08.643 35568== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:40.568 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:08.675 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:40.584 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:08.695 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:40.671 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:08.777 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:40.715 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:08.808 35566== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:40.766 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:08.876 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:40.839 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:08.930 35567== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:40.855 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:08.966 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:40.863 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:08.971 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:40.911 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:09.012 35565== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:40.932 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:09.018 35568== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:40.935 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:09.041 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.034 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:09.143 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.082 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:09.175 35566== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.119 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:09.230 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.151 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:09.259 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.152 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:09.244 35567== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.205 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:09.311 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.227 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:09.328 35565== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.253 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:09.339 35568== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.306 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:09.416 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.384 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:09.495 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.396 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:09.489 35566== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.417 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:09.508 35567== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.442 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:09.550 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.448 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:41.470 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:09.576 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.510 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:09.611 35565== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.522 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:09.608 35568== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.538+0000 7f170bf2b640 10 monclient: get_monmap_and_config 2024-03-25T19:35:41.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.538+0000 7f170bf2b640 10 monclient: build_initial_monmap 2024-03-25T19:35:41.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.538+0000 7f170bf2b640 10 monclient: monmap: 2024-03-25T19:35:41.538 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:35:41.538 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:41.539 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:35:41.539770+0000 2024-03-25T19:35:41.539 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:35:41.539770+0000 2024-03-25T19:35:41.539 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:41.539 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:41.539 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:41.539 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:35:41.539 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:35:41.539 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:41.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.539+0000 7f170bf2b640 1 Processor -- start 2024-03-25T19:35:41.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.539+0000 7f170bf2b640 1 -- start start 2024-03-25T19:35:41.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.539+0000 7f170bf2b640 10 monclient: init 2024-03-25T19:35:41.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.539+0000 7f170bf2b640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:41.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.539+0000 7f170bf2b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:35:41.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.539+0000 7f170bf2b640 10 monclient(hunting): picked mon.noname-c con 0x7f17040a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:41.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.539+0000 7f170bf2b640 10 monclient(hunting): picked mon.noname-b con 0x7f170413f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:41.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.539+0000 7f170bf2b640 10 monclient(hunting): picked mon.noname-a con 0x7f17041430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:41.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.539+0000 7f170bf2b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1704142520 con 0x7f17041430c0 2024-03-25T19:35:41.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.539+0000 7f170bf2b640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17040b28a0 con 0x7f17040a98f0 2024-03-25T19:35:41.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.539+0000 7f170bf2b640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17040b2a30 con 0x7f170413f710 2024-03-25T19:35:41.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.539+0000 7f170bf2b640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:41.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.539+0000 7f170bf2b640 10 monclient(hunting): authenticate will time out at 5405.710806s 2024-03-25T19:35:41.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.540+0000 7f1709cc7640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f17040a98f0 0x7f17040a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:47624/0 (socket says 172.21.15.44:47624) 2024-03-25T19:35:41.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.540+0000 7f17094c6640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f170413f710 0x7f17041429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:35564/0 (socket says 172.21.15.44:35564) 2024-03-25T19:35:41.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.540+0000 7f1709cc7640 1 -- 172.21.15.44:0/3056130588 learned_addr learned my addr 172.21.15.44:0/3056130588 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:35:41.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.542+0000 7f1708cc5640 1 -- 172.21.15.44:0/3056130588 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144692967 0 0) 0x7f17040b28a0 con 0x7f17040a98f0 2024-03-25T19:35:41.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.542+0000 7f1708cc5640 10 monclient(hunting): _negotiate 2024-03-25T19:35:41.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.542+0000 7f1708cc5640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:41.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.542+0000 7f1708cc5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:41.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.542+0000 7f1708cc5640 10 monclient(hunting): my global_id is 4204 2024-03-25T19:35:41.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.542+0000 7f1708cc5640 1 -- 172.21.15.44:0/3056130588 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16f0003520 con 0x7f17040a98f0 2024-03-25T19:35:41.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.543+0000 7f1708cc5640 1 -- 172.21.15.44:0/3056130588 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617156240 0 0) 0x7f17040b2a30 con 0x7f170413f710 2024-03-25T19:35:41.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.543+0000 7f1708cc5640 10 monclient(hunting): _negotiate 2024-03-25T19:35:41.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.543+0000 7f1708cc5640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:41.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.543+0000 7f1708cc5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:41.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.543+0000 7f1708cc5640 10 monclient(hunting): my global_id is 4190 2024-03-25T19:35:41.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.543+0000 7f1708cc5640 1 -- 172.21.15.44:0/3056130588 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17040b28a0 con 0x7f170413f710 2024-03-25T19:35:41.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.543+0000 7f1708cc5640 1 -- 172.21.15.44:0/3056130588 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3090013586 0 0) 0x7f1704142520 con 0x7f17041430c0 2024-03-25T19:35:41.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.543+0000 7f1708cc5640 10 monclient(hunting): _negotiate 2024-03-25T19:35:41.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.543+0000 7f1708cc5640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:41.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.543+0000 7f1708cc5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:41.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.543+0000 7f1708cc5640 10 monclient(hunting): my global_id is 4218 2024-03-25T19:35:41.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.543+0000 7f1708cc5640 1 -- 172.21.15.44:0/3056130588 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17040b2a30 con 0x7f17041430c0 2024-03-25T19:35:41.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.545+0000 7f1708cc5640 1 -- 172.21.15.44:0/3056130588 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3170925756 0 0) 0x7f16f0003520 con 0x7f17040a98f0 2024-03-25T19:35:41.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.545+0000 7f1708cc5640 1 -- 172.21.15.44:0/3056130588 >> v1:172.21.15.44:6790/0 conn(0x7f170413f710 legacy=0x7f17041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.545+0000 7f1708cc5640 1 -- 172.21.15.44:0/3056130588 >> v1:172.21.15.44:6789/0 conn(0x7f17041430c0 legacy=0x7f170414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.545+0000 7f1708cc5640 10 monclient: _finish_hunting 0 2024-03-25T19:35:41.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.545+0000 7f1708cc5640 1 monclient: found mon.noname-c 2024-03-25T19:35:41.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.545+0000 7f1708cc5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:41.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.545+0000 7f1708cc5640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:35:41.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.545+0000 7f1708cc5640 1 -- 172.21.15.44:0/3056130588 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17040b44c0 con 0x7f17040a98f0 2024-03-25T19:35:41.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.545+0000 7f1708cc5640 10 monclient: _finish_auth 0 2024-03-25T19:35:41.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.545+0000 7f1708cc5640 10 monclient: _check_auth_tickets 2024-03-25T19:35:41.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.545+0000 7f1708cc5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:41.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.545+0000 7f1708cc5640 1 -- 172.21.15.44:0/3056130588 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f1700001690 con 0x7f17040a98f0 2024-03-25T19:35:41.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.545+0000 7f170bf2b640 5 monclient: authenticate success, global_id 4204 2024-03-25T19:35:41.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.545+0000 7f170bf2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:41.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.545+0000 7f1708cc5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:41.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.545+0000 7f1708cc5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:41.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.545+0000 7f1708cc5640 10 monclient: dump: 2024-03-25T19:35:41.547 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:35:41.547 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:41.547 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:41.547 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:41.547 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:41.547 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:41.547 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:41.547 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:41.547 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:41.548 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:41.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.545+0000 7f170bf2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:41.550 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.547+0000 7f1708cc5640 1 -- 172.21.15.44:0/3056130588 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17000017f0 con 0x7f17040a98f0 2024-03-25T19:35:41.550 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.547+0000 7f1708cc5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:41.550 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.547+0000 7f1708cc5640 1 -- 172.21.15.44:0/3056130588 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f1700001c00 con 0x7f17040a98f0 2024-03-25T19:35:41.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.547+0000 7f1708cc5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:41.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.547+0000 7f1708cc5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:41.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.547+0000 7f1708cc5640 10 monclient: dump: 2024-03-25T19:35:41.551 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:35:41.551 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:41.551 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:41.551 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:41.551 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:41.551 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:41.551 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:41.551 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:41.551 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:41.551 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:41.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.547+0000 7f170bf2b640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:41.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.547+0000 7f170bf2b640 4 set_mon_vals no callback set 2024-03-25T19:35:41.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.547+0000 7f170bf2b640 10 monclient: shutdown 2024-03-25T19:35:41.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.547+0000 7f170bf2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:41.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.547+0000 7f170bf2b640 1 -- 172.21.15.44:0/3056130588 >> v1:172.21.15.111:6789/0 conn(0x7f17040a98f0 legacy=0x7f17040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.547+0000 7f170bf2b640 1 -- 172.21.15.44:0/3056130588 shutdown_connections 2024-03-25T19:35:41.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.547+0000 7f170bf2b640 1 -- 172.21.15.44:0/3056130588 >> 172.21.15.44:0/3056130588 conn(0x7f17040b7ea0 msgr2=0x7f17040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:41.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.548+0000 7f170bf2b640 1 -- 172.21.15.44:0/3056130588 shutdown_connections 2024-03-25T19:35:41.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.548+0000 7f170bf2b640 1 -- 172.21.15.44:0/3056130588 wait complete. 2024-03-25T19:35:41.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.548+0000 7f170bf2b640 10 monclient: build_initial_monmap 2024-03-25T19:35:41.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.548+0000 7f170bf2b640 10 monclient: monmap: 2024-03-25T19:35:41.552 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:35:41.552 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:41.552 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:35:41.552 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:35:41.552 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:41.552 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:35:41.552 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:41.553 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:35:41.553 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:35:41.553 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:41.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.548+0000 7f170bf2b640 1 librados: starting msgr at 2024-03-25T19:35:41.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.548+0000 7f170bf2b640 1 librados: starting objecter 2024-03-25T19:35:41.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.549+0000 7f170bf2b640 1 Processor -- start 2024-03-25T19:35:41.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.549+0000 7f170bf2b640 1 -- start start 2024-03-25T19:35:41.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.549+0000 7f170bf2b640 1 librados: setting wanted keys 2024-03-25T19:35:41.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.549+0000 7f170bf2b640 1 librados: calling monclient init 2024-03-25T19:35:41.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.549+0000 7f170bf2b640 10 monclient: init 2024-03-25T19:35:41.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.549+0000 7f170bf2b640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:41.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.549+0000 7f170bf2b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:41.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.549+0000 7f170bf2b640 10 monclient(hunting): picked mon.noname-c con 0x7f17040a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:41.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.549+0000 7f170bf2b640 10 monclient(hunting): picked mon.noname-b con 0x7f170413f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:41.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.549+0000 7f170bf2b640 10 monclient(hunting): picked mon.noname-a con 0x7f17041430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:41.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.549+0000 7f170bf2b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1704138250 con 0x7f17041430c0 2024-03-25T19:35:41.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.549+0000 7f170bf2b640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1704137d50 con 0x7f170413f710 2024-03-25T19:35:41.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.549+0000 7f170bf2b640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1704137ee0 con 0x7f17040a98f0 2024-03-25T19:35:41.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.549+0000 7f170bf2b640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:41.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.549+0000 7f170bf2b640 10 monclient(hunting): authenticate will time out at 5405.720419s 2024-03-25T19:35:41.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.549+0000 7f1709cc7640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f17040a98f0 0x7f1704137640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:35572/0 (socket says 172.21.15.44:35572) 2024-03-25T19:35:41.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.549+0000 7f1709cc7640 1 -- 172.21.15.44:0/701366222 learned_addr learned my addr 172.21.15.44:0/701366222 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:35:41.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.550+0000 7f16fa7fc640 10 client.?.objecter ms_handle_connect 0x7f17040a98f0 2024-03-25T19:35:41.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.550+0000 7f16fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:41.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.550+0000 7f16fa7fc640 10 client.?.objecter ms_handle_connect 0x7f17041430c0 2024-03-25T19:35:41.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.550+0000 7f16fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:41.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.550+0000 7f16fa7fc640 10 client.?.objecter ms_handle_connect 0x7f170413f710 2024-03-25T19:35:41.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.550+0000 7f16fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:41.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.552+0000 7f16fa7fc640 1 -- 172.21.15.44:0/701366222 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859732916 0 0) 0x7f1704137d50 con 0x7f170413f710 2024-03-25T19:35:41.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.552+0000 7f16fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:41.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.552+0000 7f16fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:41.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.552+0000 7f16fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:41.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.552+0000 7f16fa7fc640 10 monclient(hunting): my global_id is 4207 2024-03-25T19:35:41.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.552+0000 7f16fa7fc640 1 -- 172.21.15.44:0/701366222 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16e8004eb0 con 0x7f170413f710 2024-03-25T19:35:41.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.552+0000 7f16fa7fc640 1 -- 172.21.15.44:0/701366222 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1818344240 0 0) 0x7f1704137ee0 con 0x7f17040a98f0 2024-03-25T19:35:41.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.552+0000 7f16fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:41.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.552+0000 7f16fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:41.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.552+0000 7f16fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:41.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.552+0000 7f16fa7fc640 10 monclient(hunting): my global_id is 4193 2024-03-25T19:35:41.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.552+0000 7f16fa7fc640 1 -- 172.21.15.44:0/701366222 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1704137d50 con 0x7f17040a98f0 2024-03-25T19:35:41.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.552+0000 7f16fa7fc640 1 -- 172.21.15.44:0/701366222 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297294587 0 0) 0x7f1704138250 con 0x7f17041430c0 2024-03-25T19:35:41.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.552+0000 7f16fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:41.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.552+0000 7f16fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:41.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.552+0000 7f16fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:41.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.552+0000 7f16fa7fc640 10 monclient(hunting): my global_id is 4221 2024-03-25T19:35:41.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.552+0000 7f16fa7fc640 1 -- 172.21.15.44:0/701366222 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1704137ee0 con 0x7f17041430c0 2024-03-25T19:35:41.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.556+0000 7f16fa7fc640 1 -- 172.21.15.44:0/701366222 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 640406564 0 0) 0x7f16e8004eb0 con 0x7f170413f710 2024-03-25T19:35:41.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.556+0000 7f16fa7fc640 1 -- 172.21.15.44:0/701366222 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1704138250 con 0x7f170413f710 2024-03-25T19:35:41.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.556+0000 7f16fa7fc640 1 -- 172.21.15.44:0/701366222 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f16f4001970 con 0x7f170413f710 2024-03-25T19:35:41.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.556+0000 7f16fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:41.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.556+0000 7f16fa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:41.558 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.556+0000 7f16fa7fc640 10 monclient(hunting): dump: 2024-03-25T19:35:41.558 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:35:41.558 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:41.558 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:41.558 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:41.558 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:41.558 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:41.558 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:41.559 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:41.559 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:41.559 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:41.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.556+0000 7f16fa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:41.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.556+0000 7f16fa7fc640 1 -- 172.21.15.44:0/701366222 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1800521031 0 0) 0x7f1704137d50 con 0x7f17040a98f0 2024-03-25T19:35:41.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.556+0000 7f16fa7fc640 1 -- 172.21.15.44:0/701366222 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16e8004eb0 con 0x7f17040a98f0 2024-03-25T19:35:41.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.556+0000 7f16fa7fc640 1 -- 172.21.15.44:0/701366222 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f1700001b40 con 0x7f17040a98f0 2024-03-25T19:35:41.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.556+0000 7f16fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:41.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.556+0000 7f16fa7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:41.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.556+0000 7f16fa7fc640 10 monclient(hunting): dump: 2024-03-25T19:35:41.559 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:35:41.559 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:41.559 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:41.559 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:41.559 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:41.560 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:41.560 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:41.560 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:41.560 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:41.560 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:41.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.557+0000 7f16fa7fc640 1 -- 172.21.15.44:0/701366222 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2203917318 0 0) 0x7f1704137ee0 con 0x7f17041430c0 2024-03-25T19:35:41.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.557+0000 7f16fa7fc640 1 -- 172.21.15.44:0/701366222 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1704137d50 con 0x7f17041430c0 2024-03-25T19:35:41.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.557+0000 7f16fa7fc640 1 -- 172.21.15.44:0/701366222 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f16fc001bd0 con 0x7f17041430c0 2024-03-25T19:35:41.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.557+0000 7f16fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:41.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.557+0000 7f16fa7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:41.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.557+0000 7f16fa7fc640 10 monclient(hunting): dump: 2024-03-25T19:35:41.560 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:35:41.561 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:41.561 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:41.561 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:41.561 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:41.561 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:41.561 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:41.561 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:41.561 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:41.561 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:41.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.559+0000 7f16fa7fc640 1 -- 172.21.15.44:0/701366222 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 974479190 0 0) 0x7f1704138250 con 0x7f170413f710 2024-03-25T19:35:41.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.559+0000 7f16fa7fc640 1 -- 172.21.15.44:0/701366222 >> v1:172.21.15.44:6790/0 conn(0x7f17040a98f0 legacy=0x7f1704137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.559+0000 7f16fa7fc640 1 -- 172.21.15.44:0/701366222 >> v1:172.21.15.44:6789/0 conn(0x7f17041430c0 legacy=0x7f17041d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.559+0000 7f16fa7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:35:41.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.559+0000 7f16fa7fc640 1 monclient: found mon.b 2024-03-25T19:35:41.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.559+0000 7f16fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:41.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.559+0000 7f16fa7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:41.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.559+0000 7f16fa7fc640 1 -- 172.21.15.44:0/701366222 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17040b7730 con 0x7f170413f710 2024-03-25T19:35:41.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.559+0000 7f16fa7fc640 10 monclient: _finish_auth 0 2024-03-25T19:35:41.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.559+0000 7f16fa7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:35:41.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.559+0000 7f16fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:41.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.559+0000 7f170bf2b640 5 monclient: authenticate success, global_id 4207 2024-03-25T19:35:41.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.559+0000 7f170bf2b640 10 monclient: _renew_subs 2024-03-25T19:35:41.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.559+0000 7f170bf2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:41.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.559+0000 7f170bf2b640 1 -- 172.21.15.44:0/701366222 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17040b7920 con 0x7f170413f710 2024-03-25T19:35:41.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.559+0000 7f170bf2b640 10 client.4207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:41.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.559+0000 7f170bf2b640 10 monclient: _renew_subs 2024-03-25T19:35:41.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.559+0000 7f170bf2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:41.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.559+0000 7f170bf2b640 1 -- 172.21.15.44:0/701366222 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17041e33d0 con 0x7f170413f710 2024-03-25T19:35:41.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.559+0000 7f170bf2b640 1 librados: init done 2024-03-25T19:35:41.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.560+0000 7f170bf2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:41.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.560+0000 7f170bf2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:41.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.560+0000 7f170bf2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:41.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.560+0000 7f170bf2b640 1 -- 172.21.15.44:0/701366222 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17040627b0 con 0x7f170413f710 2024-03-25T19:35:41.577 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:09.686 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.660 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:09.771 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.661 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:09.754 35566== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.694 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:09.785 35567== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.736 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:09.843 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.744 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:09.850 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.774 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:09.875 35565== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.792 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:09.878 35568== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.801+0000 7f16fa7fc640 1 -- 172.21.15.44:0/701366222 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16f40047e0 con 0x7f170413f710 2024-03-25T19:35:41.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.801+0000 7f16fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:41.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.801+0000 7f16fa7fc640 1 -- 172.21.15.44:0/701366222 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f16f4004af0 con 0x7f170413f710 2024-03-25T19:35:41.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.801+0000 7f16fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:41.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.801+0000 7f16fa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:41.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.801+0000 7f16fa7fc640 10 monclient: dump: 2024-03-25T19:35:41.801 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:35:41.801 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:41.801 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:41.801 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:41.801 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:41.801 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:41.801 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:41.801 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:41.801 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:41.801 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:41.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.801+0000 7f16fb7fe640 4 set_mon_vals no callback set 2024-03-25T19:35:41.802 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.801+0000 7f16fa7fc640 1 -- 172.21.15.44:0/701366222 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f16f401aac0 con 0x7f170413f710 2024-03-25T19:35:41.802 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.801+0000 7f1709cc7640 10 monclient: get_auth_request con 0x7f16e806e0f0 auth_method 0 2024-03-25T19:35:41.802 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.802+0000 7f16fa7fc640 10 client.4207.objecter ms_handle_connect 0x7f16e806e0f0 2024-03-25T19:35:41.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.806+0000 7f16fa7fc640 1 -- 172.21.15.44:0/701366222 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f16f40546e0 con 0x7f170413f710 2024-03-25T19:35:41.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.806+0000 7f16fa7fc640 10 client.4207.objecter ms_dispatch 0x7f1704000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:41.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.806+0000 7f16fa7fc640 3 client.4207.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:41.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.806+0000 7f16fa7fc640 3 client.4207.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:41.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.806+0000 7f16fa7fc640 20 client.4207.objecter dump_active .. 0 homeless 2024-03-25T19:35:41.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.807+0000 7f16fa7fc640 1 -- 172.21.15.44:0/701366222 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f16f40501f0 con 0x7f170413f710 2024-03-25T19:35:41.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.807+0000 7f16fa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:41.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.807+0000 7f16fa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:41.841 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:09.950 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.924 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:10.035 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.925 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:10.018 35566== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.958 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:10.050 35567== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:41.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.994+0000 7f170bf2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:41.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.994+0000 7f170bf2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:41.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.994+0000 7f170bf2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:41.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:41.994+0000 7f170bf2b640 1 -- 172.21.15.44:0/701366222 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17041383e0 con 0x7f170413f710 2024-03-25T19:35:42.008 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:10.114 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.038 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:10.139 35565== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.039 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:10.145 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:42.040+0000 7f16fa7fc640 1 -- 172.21.15.44:0/701366222 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f16f4052120 con 0x7f170413f710 2024-03-25T19:35:42.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:42.040+0000 7f16fa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:42.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:42.040+0000 7f16fa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:42.041 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:35:42.041 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:42.043 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:42.042+0000 7f170bf2b640 10 librados: watch_flush enter 2024-03-25T19:35:42.043 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:42.042+0000 7f170bf2b640 10 librados: watch_flush exit 2024-03-25T19:35:42.043 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:42.042+0000 7f170bf2b640 20 client.4207.objecter shutdown clearing up homeless session... 2024-03-25T19:35:42.043 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:42.042+0000 7f170bf2b640 10 client.4207.objecter successfully canceled tick 2024-03-25T19:35:42.043 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:42.042+0000 7f170bf2b640 1 -- 172.21.15.44:0/701366222 >> v1:172.21.15.111:6800/3883215793 conn(0x7f16e806e0f0 legacy=0x7f16e80705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:42.043 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:42.042+0000 7f170bf2b640 10 monclient: shutdown 2024-03-25T19:35:42.043 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:42.042+0000 7f170bf2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:42.043 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:42.042+0000 7f170bf2b640 1 -- 172.21.15.44:0/701366222 >> v1:172.21.15.111:6789/0 conn(0x7f170413f710 legacy=0x7f17041d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:42.043 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:42.042+0000 7f170bf2b640 1 -- 172.21.15.44:0/701366222 shutdown_connections 2024-03-25T19:35:42.043 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:42.042+0000 7f170bf2b640 1 -- 172.21.15.44:0/701366222 >> 172.21.15.44:0/701366222 conn(0x7f17040b7ea0 msgr2=0x7f1704074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:42.043 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:42.042+0000 7f170bf2b640 1 -- 172.21.15.44:0/701366222 shutdown_connections 2024-03-25T19:35:42.043 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:42.042+0000 7f170bf2b640 1 -- 172.21.15.44:0/701366222 wait complete. 2024-03-25T19:35:42.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:42.042+0000 7f170bf2b640 1 librados: shutdown 2024-03-25T19:35:42.052 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:42.067 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:10.153 35568== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.109 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:10.219 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.188 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:10.299 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.194 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:10.287 35566== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.228 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:10.319 35567== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.302 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:10.403 35565== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.336 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:10.422 35568== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.341 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:10.448 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.461 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:10.554 35566== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.491 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:10.597 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.499 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:10.590 35567== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.566 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:10.667 35565== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.579 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:10.685 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.600 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:10.686 35568== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.630 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:10.740 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.680 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:10.791 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.725 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:10.818 35566== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.727 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:10.837 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.757 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:10.868 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.763 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:10.854 35567== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.830 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:10.931 35565== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.880 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:10.966 35568== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.904 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:11.012 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.959 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:11.066 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:42.989 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:11.082 35566== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:43.256 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:11.347 35567== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:43.311 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:11.403 35567== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:43.356 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:11.457 35565== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:43.402 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:11.488 35568== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:43.430 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:11.531 35565== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:43.481 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:11.567 35568== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:43.484 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:11.577 35566== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:43.540 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:11.633 35566== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:47.969 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:16.074 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:48.098 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:16.209 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:48.335 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:16.444 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:48.341 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:16.448 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:48.858 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:48.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:48.996+0000 7ff4d17eb640 10 monclient: get_monmap_and_config 2024-03-25T19:35:48.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:48.996+0000 7ff4d17eb640 10 monclient: build_initial_monmap 2024-03-25T19:35:48.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:48.996+0000 7ff4d17eb640 10 monclient: monmap: 2024-03-25T19:35:48.995 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:35:48.995 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:48.995 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:35:48.996171+0000 2024-03-25T19:35:48.995 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:35:48.996171+0000 2024-03-25T19:35:48.995 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:48.995 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:48.995 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:48.995 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:35:48.995 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:35:48.996 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:48.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:48.996+0000 7ff4d17eb640 1 Processor -- start 2024-03-25T19:35:48.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:48.996+0000 7ff4d17eb640 1 -- start start 2024-03-25T19:35:48.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:48.996+0000 7ff4d17eb640 10 monclient: init 2024-03-25T19:35:48.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:48.997+0000 7ff4d17eb640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:48.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:48.997+0000 7ff4d17eb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:35:48.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:48.997+0000 7ff4d17eb640 10 monclient(hunting): picked mon.noname-c con 0x7ff4cc0b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:48.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:48.997+0000 7ff4d17eb640 10 monclient(hunting): picked mon.noname-b con 0x7ff4cc0b1730 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:48.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:48.997+0000 7ff4d17eb640 10 monclient(hunting): picked mon.noname-a con 0x7ff4cc14be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:48.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:48.997+0000 7ff4d17eb640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4cc0af940 con 0x7ff4cc14be90 2024-03-25T19:35:48.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:48.997+0000 7ff4d17eb640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4cc0b0a70 con 0x7ff4cc0b4d60 2024-03-25T19:35:48.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:48.997+0000 7ff4d17eb640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4cc0b0d50 con 0x7ff4cc0b1730 2024-03-25T19:35:48.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:48.997+0000 7ff4d17eb640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:48.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:48.997+0000 7ff4d17eb640 10 monclient(hunting): authenticate will time out at 5413.167167s 2024-03-25T19:35:48.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:48.997+0000 7ff4ca7fc640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7ff4cc0b1730 0x7ff4cc0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:56932/0 (socket says 172.21.15.44:56932) 2024-03-25T19:35:48.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:48.997+0000 7ff4ca7fc640 1 -- 172.21.15.44:0/2434088598 learned_addr learned my addr 172.21.15.44:0/2434088598 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:35:49.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.000+0000 7ff4c9ffb640 1 -- 172.21.15.44:0/2434088598 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653601658 0 0) 0x7ff4cc0b0a70 con 0x7ff4cc0b4d60 2024-03-25T19:35:49.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.000+0000 7ff4c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.000+0000 7ff4c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.000+0000 7ff4c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.000+0000 7ff4c9ffb640 10 monclient(hunting): my global_id is 4210 2024-03-25T19:35:49.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.000+0000 7ff4c9ffb640 1 -- 172.21.15.44:0/2434088598 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4b4003520 con 0x7ff4cc0b4d60 2024-03-25T19:35:49.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.001+0000 7ff4c9ffb640 1 -- 172.21.15.44:0/2434088598 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700828961 0 0) 0x7ff4cc0b0d50 con 0x7ff4cc0b1730 2024-03-25T19:35:49.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.001+0000 7ff4c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.001+0000 7ff4c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.001+0000 7ff4c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.001+0000 7ff4c9ffb640 10 monclient(hunting): my global_id is 4196 2024-03-25T19:35:49.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.001+0000 7ff4c9ffb640 1 -- 172.21.15.44:0/2434088598 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4cc0b0a70 con 0x7ff4cc0b1730 2024-03-25T19:35:49.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.001+0000 7ff4c9ffb640 1 -- 172.21.15.44:0/2434088598 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3301742966 0 0) 0x7ff4cc0af940 con 0x7ff4cc14be90 2024-03-25T19:35:49.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.001+0000 7ff4c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.001 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.001+0000 7ff4c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.001 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.001+0000 7ff4c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.001 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.001+0000 7ff4c9ffb640 10 monclient(hunting): my global_id is 4224 2024-03-25T19:35:49.001 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.001+0000 7ff4c9ffb640 1 -- 172.21.15.44:0/2434088598 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4cc0b0d50 con 0x7ff4cc14be90 2024-03-25T19:35:49.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.004+0000 7ff4c9ffb640 1 -- 172.21.15.44:0/2434088598 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 862777192 0 0) 0x7ff4b4003520 con 0x7ff4cc0b4d60 2024-03-25T19:35:49.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.004+0000 7ff4c9ffb640 1 -- 172.21.15.44:0/2434088598 >> v1:172.21.15.44:6790/0 conn(0x7ff4cc0b1730 legacy=0x7ff4cc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.004+0000 7ff4c9ffb640 1 -- 172.21.15.44:0/2434088598 >> v1:172.21.15.44:6789/0 conn(0x7ff4cc14be90 legacy=0x7ff4cc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.004+0000 7ff4c9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:35:49.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.004+0000 7ff4c9ffb640 1 monclient: found mon.noname-c 2024-03-25T19:35:49.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.004+0000 7ff4c9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:49.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.004+0000 7ff4c9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:35:49.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.004+0000 7ff4c9ffb640 1 -- 172.21.15.44:0/2434088598 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4cc0b1030 con 0x7ff4cc0b4d60 2024-03-25T19:35:49.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.004+0000 7ff4c9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:35:49.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.004+0000 7ff4c9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:35:49.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.004+0000 7ff4c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:49.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.004+0000 7ff4d17eb640 5 monclient: authenticate success, global_id 4210 2024-03-25T19:35:49.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.004+0000 7ff4d17eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:49.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.004+0000 7ff4c9ffb640 1 -- 172.21.15.44:0/2434088598 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff4bc001690 con 0x7ff4cc0b4d60 2024-03-25T19:35:49.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.004+0000 7ff4c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.004+0000 7ff4c9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:49.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.004+0000 7ff4c9ffb640 10 monclient: dump: 2024-03-25T19:35:49.006 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:35:49.006 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:49.006 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:49.006 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:49.006 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.006 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:49.006 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:49.006 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:49.007 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:49.007 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:49.007 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.004+0000 7ff4d17eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:49.007 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.006+0000 7ff4c9ffb640 1 -- 172.21.15.44:0/2434088598 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4bc001870 con 0x7ff4cc0b4d60 2024-03-25T19:35:49.007 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.006+0000 7ff4c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:49.007 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.006+0000 7ff4d17eb640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:49.007 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.006+0000 7ff4d17eb640 4 set_mon_vals no callback set 2024-03-25T19:35:49.008 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.007+0000 7ff4c9ffb640 1 -- 172.21.15.44:0/2434088598 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff4bc001690 con 0x7ff4cc0b4d60 2024-03-25T19:35:49.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.007+0000 7ff4c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.007+0000 7ff4c9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:49.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.007+0000 7ff4c9ffb640 10 monclient: dump: 2024-03-25T19:35:49.009 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:35:49.009 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:49.009 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:49.009 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:49.009 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.009 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:49.009 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:49.009 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:49.009 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:49.009 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:49.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.007+0000 7ff4d17eb640 10 monclient: shutdown 2024-03-25T19:35:49.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.007+0000 7ff4d17eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:49.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.007+0000 7ff4d17eb640 1 -- 172.21.15.44:0/2434088598 >> v1:172.21.15.111:6789/0 conn(0x7ff4cc0b4d60 legacy=0x7ff4cc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.007+0000 7ff4d17eb640 1 -- 172.21.15.44:0/2434088598 shutdown_connections 2024-03-25T19:35:49.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.007+0000 7ff4d17eb640 1 -- 172.21.15.44:0/2434088598 >> 172.21.15.44:0/2434088598 conn(0x7ff4cc0617a0 msgr2=0x7ff4cc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:49.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.008+0000 7ff4d17eb640 1 -- 172.21.15.44:0/2434088598 shutdown_connections 2024-03-25T19:35:49.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.008+0000 7ff4d17eb640 1 -- 172.21.15.44:0/2434088598 wait complete. 2024-03-25T19:35:49.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.009+0000 7ff4d17eb640 10 monclient: build_initial_monmap 2024-03-25T19:35:49.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.009+0000 7ff4d17eb640 10 monclient: monmap: 2024-03-25T19:35:49.010 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:35:49.010 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:49.010 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:35:49.010 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:35:49.010 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:49.010 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:35:49.010 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:49.010 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:35:49.011 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:35:49.011 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:49.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.010+0000 7ff4d17eb640 1 librados: starting msgr at 2024-03-25T19:35:49.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.010+0000 7ff4d17eb640 1 librados: starting objecter 2024-03-25T19:35:49.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.010+0000 7ff4d17eb640 1 Processor -- start 2024-03-25T19:35:49.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.010+0000 7ff4d17eb640 1 -- start start 2024-03-25T19:35:49.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.010+0000 7ff4d17eb640 1 librados: setting wanted keys 2024-03-25T19:35:49.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.010+0000 7ff4d17eb640 1 librados: calling monclient init 2024-03-25T19:35:49.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.010+0000 7ff4d17eb640 10 monclient: init 2024-03-25T19:35:49.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.011+0000 7ff4d17eb640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:49.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.011+0000 7ff4d17eb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:49.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.012+0000 7ff4d17eb640 10 monclient(hunting): picked mon.noname-b con 0x7ff4cc0b1730 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:49.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.012+0000 7ff4d17eb640 10 monclient(hunting): picked mon.noname-a con 0x7ff4cc0b4d60 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:49.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.012+0000 7ff4d17eb640 10 monclient(hunting): picked mon.noname-c con 0x7ff4cc14be90 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:49.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.012+0000 7ff4d17eb640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4cc13ed30 con 0x7ff4cc0b4d60 2024-03-25T19:35:49.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.012+0000 7ff4d17eb640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4cc1307c0 con 0x7ff4cc0b1730 2024-03-25T19:35:49.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.012+0000 7ff4caffd640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff4cc0b1730 0x7ff4cc1d78a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:60220/0 (socket says 172.21.15.44:60220) 2024-03-25T19:35:49.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.012+0000 7ff4caffd640 1 -- 172.21.15.44:0/3933838008 learned_addr learned my addr 172.21.15.44:0/3933838008 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:35:49.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.012+0000 7ff4d17eb640 1 -- 172.21.15.44:0/3933838008 --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4cc1369e0 con 0x7ff4cc14be90 2024-03-25T19:35:49.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.012+0000 7ff4d17eb640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:49.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.012+0000 7ff4d17eb640 10 monclient(hunting): authenticate will time out at 5413.182794s 2024-03-25T19:35:49.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.013+0000 7ff4b37fe640 10 client.?.objecter ms_handle_connect 0x7ff4cc0b1730 2024-03-25T19:35:49.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.013+0000 7ff4b37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:49.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.013+0000 7ff4b37fe640 10 client.?.objecter ms_handle_connect 0x7ff4cc14be90 2024-03-25T19:35:49.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.013+0000 7ff4b37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:49.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.013+0000 7ff4b37fe640 10 client.?.objecter ms_handle_connect 0x7ff4cc0b4d60 2024-03-25T19:35:49.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.013+0000 7ff4b37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:49.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.016+0000 7ff4b37fe640 1 -- 172.21.15.44:0/3933838008 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626538761 0 0) 0x7ff4cc1307c0 con 0x7ff4cc0b1730 2024-03-25T19:35:49.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.016+0000 7ff4b37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.016+0000 7ff4b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.016+0000 7ff4b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.016+0000 7ff4b37fe640 10 monclient(hunting): my global_id is 4213 2024-03-25T19:35:49.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.016+0000 7ff4b37fe640 1 -- 172.21.15.44:0/3933838008 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4b8004eb0 con 0x7ff4cc0b1730 2024-03-25T19:35:49.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.016+0000 7ff4b37fe640 1 -- 172.21.15.44:0/3933838008 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766400077 0 0) 0x7ff4cc13ed30 con 0x7ff4cc0b4d60 2024-03-25T19:35:49.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.016+0000 7ff4b37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.016+0000 7ff4b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.016+0000 7ff4b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.016+0000 7ff4b37fe640 10 monclient(hunting): my global_id is 4227 2024-03-25T19:35:49.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.016+0000 7ff4b37fe640 1 -- 172.21.15.44:0/3933838008 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4cc1307c0 con 0x7ff4cc0b4d60 2024-03-25T19:35:49.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.017+0000 7ff4b37fe640 1 -- 172.21.15.44:0/3933838008 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3895233946 0 0) 0x7ff4cc1369e0 con 0x7ff4cc14be90 2024-03-25T19:35:49.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.017+0000 7ff4b37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.017+0000 7ff4b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.017+0000 7ff4b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.017+0000 7ff4b37fe640 10 monclient(hunting): my global_id is 4199 2024-03-25T19:35:49.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.017+0000 7ff4b37fe640 1 -- 172.21.15.44:0/3933838008 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4cc13ed30 con 0x7ff4cc14be90 2024-03-25T19:35:49.021 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.021+0000 7ff4b37fe640 1 -- 172.21.15.44:0/3933838008 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 594637757 0 0) 0x7ff4b8004eb0 con 0x7ff4cc0b1730 2024-03-25T19:35:49.021 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.021+0000 7ff4b37fe640 1 -- 172.21.15.44:0/3933838008 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4cc1369e0 con 0x7ff4cc0b1730 2024-03-25T19:35:49.021 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.021+0000 7ff4b37fe640 1 -- 172.21.15.44:0/3933838008 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff4bc001d60 con 0x7ff4cc0b1730 2024-03-25T19:35:49.021 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.021+0000 7ff4b37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.021 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.021+0000 7ff4b37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:49.021 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.021+0000 7ff4b37fe640 10 monclient(hunting): dump: 2024-03-25T19:35:49.021 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:35:49.021 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:49.021 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:49.021 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:49.021 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.021 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:49.021 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:49.022 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:49.022 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:49.022 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:49.022 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.021+0000 7ff4b37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:49.023 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.024+0000 7ff4b37fe640 1 -- 172.21.15.44:0/3933838008 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3658911969 0 0) 0x7ff4cc1307c0 con 0x7ff4cc0b4d60 2024-03-25T19:35:49.024 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.024+0000 7ff4b37fe640 1 -- 172.21.15.44:0/3933838008 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4b8005ad0 con 0x7ff4cc0b4d60 2024-03-25T19:35:49.024 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.024+0000 7ff4b37fe640 1 -- 172.21.15.44:0/3933838008 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff4c0001930 con 0x7ff4cc0b4d60 2024-03-25T19:35:49.024 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.024+0000 7ff4b37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.024 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.024+0000 7ff4b37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:49.024 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.024+0000 7ff4b37fe640 10 monclient(hunting): dump: 2024-03-25T19:35:49.024 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:35:49.024 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:49.024 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:49.024 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:49.024 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.024 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:49.024 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:49.024 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:49.024 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:49.025 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:49.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.024+0000 7ff4b37fe640 1 -- 172.21.15.44:0/3933838008 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2384631845 0 0) 0x7ff4cc13ed30 con 0x7ff4cc14be90 2024-03-25T19:35:49.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.024+0000 7ff4b37fe640 1 -- 172.21.15.44:0/3933838008 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4cc1307c0 con 0x7ff4cc14be90 2024-03-25T19:35:49.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.024+0000 7ff4b37fe640 1 -- 172.21.15.44:0/3933838008 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff4c4001bd0 con 0x7ff4cc14be90 2024-03-25T19:35:49.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.024+0000 7ff4b37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.024+0000 7ff4b37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:49.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.024+0000 7ff4b37fe640 10 monclient(hunting): dump: 2024-03-25T19:35:49.025 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:35:49.025 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:49.025 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:49.025 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:49.025 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.025 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:49.025 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:49.025 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:49.025 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:49.025 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:49.027 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.027+0000 7ff4b37fe640 1 -- 172.21.15.44:0/3933838008 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3825215983 0 0) 0x7ff4cc1369e0 con 0x7ff4cc0b1730 2024-03-25T19:35:49.028 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.027+0000 7ff4b37fe640 1 -- 172.21.15.44:0/3933838008 >> v1:172.21.15.44:6790/0 conn(0x7ff4cc14be90 legacy=0x7ff4cc1300b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.028 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.027+0000 7ff4b37fe640 1 -- 172.21.15.44:0/3933838008 >> v1:172.21.15.44:6789/0 conn(0x7ff4cc0b4d60 legacy=0x7ff4cc12f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.028 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.027+0000 7ff4b37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:35:49.028 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.027+0000 7ff4b37fe640 1 monclient: found mon.b 2024-03-25T19:35:49.028 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.027+0000 7ff4b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:49.028 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.027+0000 7ff4b37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:49.028 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.027+0000 7ff4b37fe640 1 -- 172.21.15.44:0/3933838008 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4cc136b70 con 0x7ff4cc0b1730 2024-03-25T19:35:49.028 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.027+0000 7ff4b37fe640 10 monclient: _finish_auth 0 2024-03-25T19:35:49.028 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.027+0000 7ff4b37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:35:49.028 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.027+0000 7ff4b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:49.028 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.028+0000 7ff4d17eb640 5 monclient: authenticate success, global_id 4213 2024-03-25T19:35:49.028 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.028+0000 7ff4d17eb640 10 monclient: _renew_subs 2024-03-25T19:35:49.028 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.028+0000 7ff4d17eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:49.028 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.028+0000 7ff4d17eb640 1 -- 172.21.15.44:0/3933838008 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4cc13e680 con 0x7ff4cc0b1730 2024-03-25T19:35:49.028 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.028+0000 7ff4d17eb640 10 client.4213.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:49.028 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.028+0000 7ff4d17eb640 10 monclient: _renew_subs 2024-03-25T19:35:49.029 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.028+0000 7ff4d17eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:49.029 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.028+0000 7ff4d17eb640 1 -- 172.21.15.44:0/3933838008 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4cc1d7fb0 con 0x7ff4cc0b1730 2024-03-25T19:35:49.029 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.028+0000 7ff4d17eb640 1 librados: init done 2024-03-25T19:35:49.029 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.030+0000 7ff4d17eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:49.029 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.030+0000 7ff4d17eb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:49.029 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.030+0000 7ff4d17eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:49.029 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.030+0000 7ff4d17eb640 1 -- 172.21.15.44:0/3933838008 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4cc136d60 con 0x7ff4cc0b1730 2024-03-25T19:35:49.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.051+0000 7ff4b37fe640 1 -- 172.21.15.44:0/3933838008 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4bc001240 con 0x7ff4cc0b1730 2024-03-25T19:35:49.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.051+0000 7ff4b37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:49.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.051+0000 7ff4b37fe640 1 -- 172.21.15.44:0/3933838008 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff4bc001d60 con 0x7ff4cc0b1730 2024-03-25T19:35:49.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.051+0000 7ff4b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.051+0000 7ff4b37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:49.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.051+0000 7ff4b37fe640 10 monclient: dump: 2024-03-25T19:35:49.050 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:35:49.050 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:49.050 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:49.050 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:49.050 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.050 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:49.051 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:49.051 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:49.051 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:49.051 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:49.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.051+0000 7ff4c97fa640 4 set_mon_vals no callback set 2024-03-25T19:35:49.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.052+0000 7ff4b37fe640 1 -- 172.21.15.44:0/3933838008 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7ff4bc0039e0 con 0x7ff4cc0b1730 2024-03-25T19:35:49.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.053+0000 7ff4ca7fc640 10 monclient: get_auth_request con 0x7ff4b806e570 auth_method 0 2024-03-25T19:35:49.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.053+0000 7ff4b37fe640 10 client.4213.objecter ms_handle_connect 0x7ff4b806e570 2024-03-25T19:35:49.054 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.055+0000 7ff4b37fe640 1 -- 172.21.15.44:0/3933838008 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7ff4bc054620 con 0x7ff4cc0b1730 2024-03-25T19:35:49.054 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.055+0000 7ff4b37fe640 10 client.4213.objecter ms_dispatch 0x7ff4cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:49.054 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.055+0000 7ff4b37fe640 3 client.4213.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:49.055 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.055+0000 7ff4b37fe640 3 client.4213.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:49.055 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.055+0000 7ff4b37fe640 20 client.4213.objecter dump_active .. 0 homeless 2024-03-25T19:35:49.100 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:17.191 35567== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:49.211 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:17.297 35568== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:49.273 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:17.366 35566== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:49.288 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.289+0000 7ff4b37fe640 1 -- 172.21.15.44:0/3933838008 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff4bc04ff60 con 0x7ff4cc0b1730 2024-03-25T19:35:49.288 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.289+0000 7ff4b37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:49.288 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.289+0000 7ff4b37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:49.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.495+0000 7ff4d17eb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:49.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.495+0000 7ff4d17eb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:49.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.495+0000 7ff4d17eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:49.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.495+0000 7ff4d17eb640 1 -- 172.21.15.44:0/3933838008 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4cc0627b0 con 0x7ff4cc0b1730 2024-03-25T19:35:49.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.509+0000 7ff4b37fe640 1 -- 172.21.15.44:0/3933838008 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7ff4bc052060 con 0x7ff4cc0b1730 2024-03-25T19:35:49.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.509+0000 7ff4b37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:49.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.509+0000 7ff4b37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:49.508 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:35:49.508 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:49.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.512+0000 7ff4d17eb640 10 librados: watch_flush enter 2024-03-25T19:35:49.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.512+0000 7ff4d17eb640 10 librados: watch_flush exit 2024-03-25T19:35:49.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.512+0000 7ff4d17eb640 20 client.4213.objecter shutdown clearing up homeless session... 2024-03-25T19:35:49.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.512+0000 7ff4d17eb640 10 client.4213.objecter successfully canceled tick 2024-03-25T19:35:49.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.512+0000 7ff4d17eb640 1 -- 172.21.15.44:0/3933838008 >> v1:172.21.15.111:6800/3883215793 conn(0x7ff4b806e570 legacy=0x7ff4b8070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.512+0000 7ff4d17eb640 10 monclient: shutdown 2024-03-25T19:35:49.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.512+0000 7ff4d17eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:49.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.512+0000 7ff4d17eb640 1 -- 172.21.15.44:0/3933838008 >> v1:172.21.15.111:6789/0 conn(0x7ff4cc0b1730 legacy=0x7ff4cc1d78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.512+0000 7ff4d17eb640 1 -- 172.21.15.44:0/3933838008 shutdown_connections 2024-03-25T19:35:49.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.512+0000 7ff4d17eb640 1 -- 172.21.15.44:0/3933838008 >> 172.21.15.44:0/3933838008 conn(0x7ff4cc0617a0 msgr2=0x7ff4cc0b6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:49.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.512+0000 7ff4d17eb640 1 -- 172.21.15.44:0/3933838008 shutdown_connections 2024-03-25T19:35:49.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.512+0000 7ff4d17eb640 1 -- 172.21.15.44:0/3933838008 wait complete. 2024-03-25T19:35:49.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:49.513+0000 7ff4d17eb640 1 librados: shutdown 2024-03-25T19:35:49.525 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:49.629 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:17.730 35565== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:53.276 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:21.382 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:53.282 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:21.388 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:53.372 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:21.483 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:53.376 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:21.487 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:53.947 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:22.054 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:53.951 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:22.058 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:53.986 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:22.095 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:53.992 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:22.101 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:54.560 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:22.653 35566== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:54.564 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:22.657 35566== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:54.683 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:22.774 35567== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:54.688 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:22.779 35567== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:54.948 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:23.034 35568== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:54.954 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:23.040 35568== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:55.185 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:23.286 35565== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:55.191 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:23.292 35565== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:56.332 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:56.451 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.451+0000 7f33b8510640 10 monclient: get_monmap_and_config 2024-03-25T19:35:56.451 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.451+0000 7f33b8510640 10 monclient: build_initial_monmap 2024-03-25T19:35:56.451 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.451+0000 7f33b8510640 10 monclient: monmap: 2024-03-25T19:35:56.451 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:35:56.451 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:56.451 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:35:56.452366+0000 2024-03-25T19:35:56.451 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:35:56.452366+0000 2024-03-25T19:35:56.451 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:56.451 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:56.451 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:56.452 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:35:56.452 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:35:56.452 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:56.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.451+0000 7f33b8510640 1 Processor -- start 2024-03-25T19:35:56.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.451+0000 7f33b8510640 1 -- start start 2024-03-25T19:35:56.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.451+0000 7f33b8510640 10 monclient: init 2024-03-25T19:35:56.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.452+0000 7f33b8510640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:56.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.452+0000 7f33b8510640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:35:56.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.452+0000 7f33b8510640 10 monclient(hunting): picked mon.noname-a con 0x7f33b00b4d60 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:56.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.452+0000 7f33b8510640 10 monclient(hunting): picked mon.noname-c con 0x7f33b00b1730 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:56.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.452+0000 7f33b8510640 10 monclient(hunting): picked mon.noname-b con 0x7f33b014be90 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:56.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.452+0000 7f33b8510640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33b00af940 con 0x7f33b00b4d60 2024-03-25T19:35:56.454 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.452+0000 7f33b8510640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33b00b0a70 con 0x7f33b00b1730 2024-03-25T19:35:56.454 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.452+0000 7f33b8510640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33b00b0d50 con 0x7f33b014be90 2024-03-25T19:35:56.454 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.452+0000 7f33b8510640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:56.454 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.452+0000 7f33b8510640 10 monclient(hunting): authenticate will time out at 5420.623781s 2024-03-25T19:35:56.454 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.453+0000 7f33b5aab640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f33b00b1730 0x7f33b00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:60224/0 (socket says 172.21.15.44:60224) 2024-03-25T19:35:56.454 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.453+0000 7f33b5aab640 1 -- 172.21.15.44:0/1553871509 learned_addr learned my addr 172.21.15.44:0/1553871509 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:35:56.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.455+0000 7f33b52aa640 1 -- 172.21.15.44:0/1553871509 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 666961921 0 0) 0x7f33b00b0a70 con 0x7f33b00b1730 2024-03-25T19:35:56.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.455+0000 7f33b52aa640 10 monclient(hunting): _negotiate 2024-03-25T19:35:56.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.455+0000 7f33b52aa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:56.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.455+0000 7f33b52aa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:56.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.455+0000 7f33b52aa640 10 monclient(hunting): my global_id is 4216 2024-03-25T19:35:56.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.455+0000 7f33b52aa640 1 -- 172.21.15.44:0/1553871509 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f339c003520 con 0x7f33b00b1730 2024-03-25T19:35:56.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.456+0000 7f33b52aa640 1 -- 172.21.15.44:0/1553871509 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 711747953 0 0) 0x7f33b00b0d50 con 0x7f33b014be90 2024-03-25T19:35:56.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.456+0000 7f33b52aa640 10 monclient(hunting): _negotiate 2024-03-25T19:35:56.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.456+0000 7f33b52aa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:56.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.456+0000 7f33b52aa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:56.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.456+0000 7f33b52aa640 10 monclient(hunting): my global_id is 4202 2024-03-25T19:35:56.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.456+0000 7f33b52aa640 1 -- 172.21.15.44:0/1553871509 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33b00b0a70 con 0x7f33b014be90 2024-03-25T19:35:56.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.456+0000 7f33b52aa640 1 -- 172.21.15.44:0/1553871509 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3204604722 0 0) 0x7f33b00af940 con 0x7f33b00b4d60 2024-03-25T19:35:56.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.456+0000 7f33b52aa640 10 monclient(hunting): _negotiate 2024-03-25T19:35:56.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.456+0000 7f33b52aa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:56.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.456+0000 7f33b52aa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:56.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.456+0000 7f33b52aa640 10 monclient(hunting): my global_id is 4230 2024-03-25T19:35:56.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.456+0000 7f33b52aa640 1 -- 172.21.15.44:0/1553871509 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33b00b0d50 con 0x7f33b00b4d60 2024-03-25T19:35:56.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.459+0000 7f33b52aa640 1 -- 172.21.15.44:0/1553871509 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2330927523 0 0) 0x7f33b00b0d50 con 0x7f33b00b4d60 2024-03-25T19:35:56.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.459+0000 7f33b52aa640 1 -- 172.21.15.44:0/1553871509 >> v1:172.21.15.44:6790/0 conn(0x7f33b014be90 legacy=0x7f33b014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:56.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.459+0000 7f33b52aa640 1 -- 172.21.15.44:0/1553871509 >> v1:172.21.15.111:6789/0 conn(0x7f33b00b1730 legacy=0x7f33b00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:56.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.459+0000 7f33b52aa640 10 monclient: _finish_hunting 0 2024-03-25T19:35:56.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.459+0000 7f33b52aa640 1 monclient: found mon.noname-a 2024-03-25T19:35:56.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.459+0000 7f33b52aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:56.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.459+0000 7f33b52aa640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:35:56.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.459+0000 7f33b52aa640 1 -- 172.21.15.44:0/1553871509 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b00b1030 con 0x7f33b00b4d60 2024-03-25T19:35:56.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.459+0000 7f33b52aa640 10 monclient: _finish_auth 0 2024-03-25T19:35:56.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.459+0000 7f33b52aa640 10 monclient: _check_auth_tickets 2024-03-25T19:35:56.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.459+0000 7f33b52aa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:56.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.459+0000 7f33b8510640 5 monclient: authenticate success, global_id 4230 2024-03-25T19:35:56.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.459+0000 7f33b8510640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:56.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.460+0000 7f33b52aa640 1 -- 172.21.15.44:0/1553871509 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f33ac0014f0 con 0x7f33b00b4d60 2024-03-25T19:35:56.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.460+0000 7f33b52aa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:56.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.460+0000 7f33b52aa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:56.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.460+0000 7f33b52aa640 10 monclient: dump: 2024-03-25T19:35:56.461 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:35:56.461 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:56.461 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:56.461 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:56.461 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:56.461 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:56.461 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:56.461 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:56.461 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:56.461 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:56.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.460+0000 7f33b8510640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:56.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.461+0000 7f33b52aa640 1 -- 172.21.15.44:0/1553871509 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33ac0017c0 con 0x7f33b00b4d60 2024-03-25T19:35:56.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.461+0000 7f33b52aa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:56.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.461+0000 7f33b8510640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:56.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.461+0000 7f33b8510640 4 set_mon_vals no callback set 2024-03-25T19:35:56.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.461+0000 7f33b52aa640 1 -- 172.21.15.44:0/1553871509 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f33ac0014f0 con 0x7f33b00b4d60 2024-03-25T19:35:56.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.461+0000 7f33b52aa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:56.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.461+0000 7f33b52aa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:56.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.461+0000 7f33b52aa640 10 monclient: dump: 2024-03-25T19:35:56.463 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:35:56.463 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:56.464 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:56.464 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:56.464 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:56.464 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:56.464 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:56.464 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:56.464 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:56.464 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:56.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.462+0000 7f33b8510640 10 monclient: shutdown 2024-03-25T19:35:56.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.462+0000 7f33b8510640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:56.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.462+0000 7f33b8510640 1 -- 172.21.15.44:0/1553871509 >> v1:172.21.15.44:6789/0 conn(0x7f33b00b4d60 legacy=0x7f33b013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:56.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.462+0000 7f33b8510640 1 -- 172.21.15.44:0/1553871509 shutdown_connections 2024-03-25T19:35:56.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.462+0000 7f33b8510640 1 -- 172.21.15.44:0/1553871509 >> 172.21.15.44:0/1553871509 conn(0x7f33b00617a0 msgr2=0x7f33b0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:56.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.462+0000 7f33b8510640 1 -- 172.21.15.44:0/1553871509 shutdown_connections 2024-03-25T19:35:56.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.463+0000 7f33b8510640 1 -- 172.21.15.44:0/1553871509 wait complete. 2024-03-25T19:35:56.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.463+0000 7f33b8510640 10 monclient: build_initial_monmap 2024-03-25T19:35:56.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.463+0000 7f33b8510640 10 monclient: monmap: 2024-03-25T19:35:56.467 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:35:56.467 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:56.467 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:35:56.467 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:35:56.467 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:56.467 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:35:56.467 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:35:56.468 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:35:56.468 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:35:56.468 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:56.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.464+0000 7f33b8510640 1 librados: starting msgr at 2024-03-25T19:35:56.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.464+0000 7f33b8510640 1 librados: starting objecter 2024-03-25T19:35:56.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.464+0000 7f33b8510640 1 Processor -- start 2024-03-25T19:35:56.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.464+0000 7f33b8510640 1 -- start start 2024-03-25T19:35:56.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.464+0000 7f33b8510640 1 librados: setting wanted keys 2024-03-25T19:35:56.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.464+0000 7f33b8510640 1 librados: calling monclient init 2024-03-25T19:35:56.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.464+0000 7f33b8510640 10 monclient: init 2024-03-25T19:35:56.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.465+0000 7f33b8510640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:56.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.465+0000 7f33b8510640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:35:56.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.465+0000 7f33b8510640 10 monclient(hunting): picked mon.noname-b con 0x7f33b00b1730 addr v1:172.21.15.111:6789/0 2024-03-25T19:35:56.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.465+0000 7f33b8510640 10 monclient(hunting): picked mon.noname-c con 0x7f33b00b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T19:35:56.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.465+0000 7f33b8510640 10 monclient(hunting): picked mon.noname-a con 0x7f33b014be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:35:56.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.465+0000 7f33b8510640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33b0134f90 con 0x7f33b014be90 2024-03-25T19:35:56.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.465+0000 7f33b8510640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33b0139ea0 con 0x7f33b00b1730 2024-03-25T19:35:56.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.465+0000 7f33b8510640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33b013a180 con 0x7f33b00b4d60 2024-03-25T19:35:56.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.465+0000 7f33b8510640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:56.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.465+0000 7f33b8510640 10 monclient(hunting): authenticate will time out at 5420.636112s 2024-03-25T19:35:56.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.465+0000 7f33b6aad640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f33b014be90 0x7f33b0134880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:39424/0 (socket says 172.21.15.44:39424) 2024-03-25T19:35:56.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.465+0000 7f33b6aad640 1 -- 172.21.15.44:0/4167860975 learned_addr learned my addr 172.21.15.44:0/4167860975 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:35:56.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.465+0000 7f33a6ffd640 10 client.?.objecter ms_handle_connect 0x7f33b014be90 2024-03-25T19:35:56.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.466+0000 7f33a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:56.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.466+0000 7f33a6ffd640 10 client.?.objecter ms_handle_connect 0x7f33b00b1730 2024-03-25T19:35:56.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.466+0000 7f33a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:56.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.467+0000 7f33a6ffd640 10 client.?.objecter ms_handle_connect 0x7f33b00b4d60 2024-03-25T19:35:56.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.467+0000 7f33a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:56.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.468+0000 7f33a6ffd640 1 -- 172.21.15.44:0/4167860975 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3243154748 0 0) 0x7f33b0139ea0 con 0x7f33b00b1730 2024-03-25T19:35:56.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.468+0000 7f33a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:56.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.468+0000 7f33a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:56.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.468+0000 7f33a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:56.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.468+0000 7f33a6ffd640 10 monclient(hunting): my global_id is 4219 2024-03-25T19:35:56.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.468+0000 7f33a6ffd640 1 -- 172.21.15.44:0/4167860975 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3394004eb0 con 0x7f33b00b1730 2024-03-25T19:35:56.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.468+0000 7f33a6ffd640 1 -- 172.21.15.44:0/4167860975 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2388546986 0 0) 0x7f33b0134f90 con 0x7f33b014be90 2024-03-25T19:35:56.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.468+0000 7f33a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:56.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.468+0000 7f33a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:56.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.468+0000 7f33a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:56.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.468+0000 7f33a6ffd640 10 monclient(hunting): my global_id is 4233 2024-03-25T19:35:56.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.468+0000 7f33a6ffd640 1 -- 172.21.15.44:0/4167860975 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33b0139ea0 con 0x7f33b014be90 2024-03-25T19:35:56.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.469+0000 7f33a6ffd640 1 -- 172.21.15.44:0/4167860975 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2207189082 0 0) 0x7f33b013a180 con 0x7f33b00b4d60 2024-03-25T19:35:56.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.469+0000 7f33a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:56.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.469+0000 7f33a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:56.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.469+0000 7f33a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:56.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.469+0000 7f33a6ffd640 10 monclient(hunting): my global_id is 4205 2024-03-25T19:35:56.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.469+0000 7f33a6ffd640 1 -- 172.21.15.44:0/4167860975 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33b0134f90 con 0x7f33b00b4d60 2024-03-25T19:35:56.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.472+0000 7f33a6ffd640 1 -- 172.21.15.44:0/4167860975 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1031011903 0 0) 0x7f3394004eb0 con 0x7f33b00b1730 2024-03-25T19:35:56.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.472+0000 7f33a6ffd640 1 -- 172.21.15.44:0/4167860975 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33b013a180 con 0x7f33b00b1730 2024-03-25T19:35:56.473 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.472+0000 7f33a6ffd640 1 -- 172.21.15.44:0/4167860975 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f33ac0014f0 con 0x7f33b00b1730 2024-03-25T19:35:56.473 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.472+0000 7f33a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:56.473 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.472+0000 7f33a6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:56.473 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.472+0000 7f33a6ffd640 10 monclient(hunting): dump: 2024-03-25T19:35:56.473 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:35:56.473 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:56.473 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:56.473 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:56.473 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:56.473 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:56.473 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:56.473 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:56.473 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:56.474 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:56.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.472+0000 7f33a6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:56.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.475+0000 7f33a6ffd640 1 -- 172.21.15.44:0/4167860975 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 672283824 0 0) 0x7f33b013a180 con 0x7f33b00b1730 2024-03-25T19:35:56.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.475+0000 7f33a6ffd640 1 -- 172.21.15.44:0/4167860975 >> v1:172.21.15.44:6790/0 conn(0x7f33b00b4d60 legacy=0x7f33b0134170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:56.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.476+0000 7f33a6ffd640 1 -- 172.21.15.44:0/4167860975 >> v1:172.21.15.44:6789/0 conn(0x7f33b014be90 legacy=0x7f33b0134880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:56.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.476+0000 7f33a6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:35:56.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.476+0000 7f33a6ffd640 1 monclient: found mon.b 2024-03-25T19:35:56.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.476+0000 7f33a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:56.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.476+0000 7f33a6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:56.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.476+0000 7f33a6ffd640 1 -- 172.21.15.44:0/4167860975 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b0142f10 con 0x7f33b00b1730 2024-03-25T19:35:56.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.476+0000 7f33a6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:35:56.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.476+0000 7f33a6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:35:56.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.476+0000 7f33a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:56.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.476+0000 7f33b8510640 5 monclient: authenticate success, global_id 4219 2024-03-25T19:35:56.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.476+0000 7f33b8510640 10 monclient: _renew_subs 2024-03-25T19:35:56.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.476+0000 7f33b8510640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:56.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.476+0000 7f33b8510640 1 -- 172.21.15.44:0/4167860975 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33b0140a90 con 0x7f33b00b1730 2024-03-25T19:35:56.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.476+0000 7f33b8510640 10 client.4219.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:56.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.476+0000 7f33b8510640 10 monclient: _renew_subs 2024-03-25T19:35:56.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.476+0000 7f33b8510640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:56.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.476+0000 7f33b8510640 1 -- 172.21.15.44:0/4167860975 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b013bef0 con 0x7f33b00b1730 2024-03-25T19:35:56.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.476+0000 7f33b8510640 1 librados: init done 2024-03-25T19:35:56.478 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.477+0000 7f33b8510640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:56.478 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.477+0000 7f33b8510640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:56.478 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.477+0000 7f33b8510640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:56.478 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.477+0000 7f33b8510640 1 -- 172.21.15.44:0/4167860975 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33b00afbd0 con 0x7f33b00b1730 2024-03-25T19:35:56.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.720+0000 7f33a6ffd640 1 -- 172.21.15.44:0/4167860975 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33ac0038c0 con 0x7f33b00b1730 2024-03-25T19:35:56.721 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.720+0000 7f33a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:56.721 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.720+0000 7f33a6ffd640 1 -- 172.21.15.44:0/4167860975 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f33ac003bd0 con 0x7f33b00b1730 2024-03-25T19:35:56.721 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.721+0000 7f33a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:56.721 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.721+0000 7f33a6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:56.721 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.721+0000 7f33a6ffd640 10 monclient: dump: 2024-03-25T19:35:56.721 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:35:56.721 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:35:56.721 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:56.721 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:35:56.721 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:56.721 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:35:56.721 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:35:56.721 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:35:56.722 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:35:56.722 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:35:56.722 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.721+0000 7f33b4aa9640 4 set_mon_vals no callback set 2024-03-25T19:35:56.722 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.722+0000 7f33a6ffd640 1 -- 172.21.15.44:0/4167860975 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f33ac019ba0 con 0x7f33b00b1730 2024-03-25T19:35:56.722 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.723+0000 7f33b5aab640 10 monclient: get_auth_request con 0x7f339406ddf0 auth_method 0 2024-03-25T19:35:56.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.723+0000 7f33a6ffd640 10 client.4219.objecter ms_handle_connect 0x7f339406ddf0 2024-03-25T19:35:56.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.726+0000 7f33a6ffd640 1 -- 172.21.15.44:0/4167860975 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f33ac053760 con 0x7f33b00b1730 2024-03-25T19:35:56.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.726+0000 7f33a6ffd640 10 client.4219.objecter ms_dispatch 0x7f33b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:56.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.726+0000 7f33a6ffd640 3 client.4219.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:56.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.726+0000 7f33a6ffd640 3 client.4219.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:56.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.726+0000 7f33a6ffd640 20 client.4219.objecter dump_active .. 0 homeless 2024-03-25T19:35:56.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.727+0000 7f33a6ffd640 1 -- 172.21.15.44:0/4167860975 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f33ac04f0a0 con 0x7f33b00b1730 2024-03-25T19:35:56.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.727+0000 7f33a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:56.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.727+0000 7f33a6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:56.962 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.961+0000 7f33b8510640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:56.962 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.962+0000 7f33b8510640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:56.962 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.962+0000 7f33b8510640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:35:56.962 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.962+0000 7f33b8510640 1 -- 172.21.15.44:0/4167860975 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33b0143380 con 0x7f33b00b1730 2024-03-25T19:35:56.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.985+0000 7f33a6ffd640 1 -- 172.21.15.44:0/4167860975 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f33ac0511a0 con 0x7f33b00b1730 2024-03-25T19:35:56.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.985+0000 7f33a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:56.985 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.985+0000 7f33a6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:56.985 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:35:56.986 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:56.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.987+0000 7f33b8510640 10 librados: watch_flush enter 2024-03-25T19:35:56.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.988+0000 7f33b8510640 10 librados: watch_flush exit 2024-03-25T19:35:56.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.988+0000 7f33b8510640 20 client.4219.objecter shutdown clearing up homeless session... 2024-03-25T19:35:56.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.988+0000 7f33b8510640 10 client.4219.objecter successfully canceled tick 2024-03-25T19:35:56.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.988+0000 7f33b8510640 1 -- 172.21.15.44:0/4167860975 >> v1:172.21.15.111:6800/3883215793 conn(0x7f339406ddf0 legacy=0x7f33940702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:56.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.988+0000 7f33b8510640 10 monclient: shutdown 2024-03-25T19:35:56.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.988+0000 7f33b8510640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:56.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.988+0000 7f33b8510640 1 -- 172.21.15.44:0/4167860975 >> v1:172.21.15.111:6789/0 conn(0x7f33b00b1730 legacy=0x7f33b0140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:56.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.988+0000 7f33b8510640 1 -- 172.21.15.44:0/4167860975 shutdown_connections 2024-03-25T19:35:56.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.988+0000 7f33b8510640 1 -- 172.21.15.44:0/4167860975 >> 172.21.15.44:0/4167860975 conn(0x7f33b00617a0 msgr2=0x7f33b00b63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:56.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.988+0000 7f33b8510640 1 -- 172.21.15.44:0/4167860975 shutdown_connections 2024-03-25T19:35:56.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.988+0000 7f33b8510640 1 -- 172.21.15.44:0/4167860975 wait complete. 2024-03-25T19:35:56.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:35:56.988+0000 7f33b8510640 1 librados: shutdown 2024-03-25T19:35:57.001 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:57.764 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:00:25.875 35554== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:58.038 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:00:26.143 35557== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:58.305 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:00:26.414 35555== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:58.591 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:00:26.698 35556== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:58.824 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:00:26.917 35566== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:59.112 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:00:27.203 35567== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:59.277 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:00:27.363 35568== Warning: unimplemented fcntl command: 1038 2024-03-25T19:35:59.569 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:00:27.670 35565== Warning: unimplemented fcntl command: 1038 2024-03-25T19:36:03.807 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:03.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.953+0000 7f6d158f1640 10 monclient: get_monmap_and_config 2024-03-25T19:36:03.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.953+0000 7f6d158f1640 10 monclient: build_initial_monmap 2024-03-25T19:36:03.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.953+0000 7f6d158f1640 10 monclient: monmap: 2024-03-25T19:36:03.953 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:36:03.953 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:03.953 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:36:03.954228+0000 2024-03-25T19:36:03.953 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:36:03.954228+0000 2024-03-25T19:36:03.953 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:03.953 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:03.954 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:36:03.954 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:36:03.954 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:36:03.954 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:03.955 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.954+0000 7f6d158f1640 1 Processor -- start 2024-03-25T19:36:03.955 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.954+0000 7f6d158f1640 1 -- start start 2024-03-25T19:36:03.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.954+0000 7f6d158f1640 10 monclient: init 2024-03-25T19:36:03.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.954+0000 7f6d158f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:03.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.954+0000 7f6d158f1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:36:03.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.954+0000 7f6d158f1640 10 monclient(hunting): picked mon.noname-a con 0x7f6d100a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:36:03.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.954+0000 7f6d158f1640 10 monclient(hunting): picked mon.noname-c con 0x7f6d1013f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:36:03.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.954+0000 7f6d158f1640 10 monclient(hunting): picked mon.noname-b con 0x7f6d101430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:36:03.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.954+0000 7f6d158f1640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d10142520 con 0x7f6d100a98f0 2024-03-25T19:36:03.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.954+0000 7f6d158f1640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d100b28a0 con 0x7f6d1013f710 2024-03-25T19:36:03.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.954+0000 7f6d158f1640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d100b2a30 con 0x7f6d101430c0 2024-03-25T19:36:03.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.954+0000 7f6d158f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:03.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.954+0000 7f6d158f1640 10 monclient(hunting): authenticate will time out at 5428.125807s 2024-03-25T19:36:03.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.955+0000 7f6d0effd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f6d100a98f0 0x7f6d100a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54016/0 (socket says 172.21.15.44:54016) 2024-03-25T19:36:03.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.955+0000 7f6d0effd640 1 -- 172.21.15.44:0/1014040821 learned_addr learned my addr 172.21.15.44:0/1014040821 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:36:03.960 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.959+0000 7f6d0e7fc640 1 -- 172.21.15.44:0/1014040821 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066008327 0 0) 0x7f6d100b2a30 con 0x7f6d101430c0 2024-03-25T19:36:03.960 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.959+0000 7f6d0e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:03.960 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.959+0000 7f6d0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:03.960 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.959+0000 7f6d0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:03.960 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.959+0000 7f6d0e7fc640 10 monclient(hunting): my global_id is 4208 2024-03-25T19:36:03.960 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.959+0000 7f6d0e7fc640 1 -- 172.21.15.44:0/1014040821 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cfc003520 con 0x7f6d101430c0 2024-03-25T19:36:03.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.959+0000 7f6d0e7fc640 1 -- 172.21.15.44:0/1014040821 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891395087 0 0) 0x7f6d100b28a0 con 0x7f6d1013f710 2024-03-25T19:36:03.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.959+0000 7f6d0e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:03.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.959+0000 7f6d0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:03.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.959+0000 7f6d0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:03.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.959+0000 7f6d0e7fc640 10 monclient(hunting): my global_id is 4222 2024-03-25T19:36:03.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.959+0000 7f6d0e7fc640 1 -- 172.21.15.44:0/1014040821 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d100b2a30 con 0x7f6d1013f710 2024-03-25T19:36:03.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.959+0000 7f6d0e7fc640 1 -- 172.21.15.44:0/1014040821 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398108200 0 0) 0x7f6d10142520 con 0x7f6d100a98f0 2024-03-25T19:36:03.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.959+0000 7f6d0e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:03.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.959+0000 7f6d0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:03.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.959+0000 7f6d0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:03.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.959+0000 7f6d0e7fc640 10 monclient(hunting): my global_id is 4236 2024-03-25T19:36:03.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.959+0000 7f6d0e7fc640 1 -- 172.21.15.44:0/1014040821 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d100b28a0 con 0x7f6d100a98f0 2024-03-25T19:36:03.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.962+0000 7f6d0e7fc640 1 -- 172.21.15.44:0/1014040821 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 698539533 0 0) 0x7f6cfc003520 con 0x7f6d101430c0 2024-03-25T19:36:03.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.962+0000 7f6d0e7fc640 1 -- 172.21.15.44:0/1014040821 >> v1:172.21.15.111:6789/0 conn(0x7f6d1013f710 legacy=0x7f6d101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:03.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.962+0000 7f6d0e7fc640 1 -- 172.21.15.44:0/1014040821 >> v1:172.21.15.44:6789/0 conn(0x7f6d100a98f0 legacy=0x7f6d100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:03.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.962+0000 7f6d0e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:36:03.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.962+0000 7f6d0e7fc640 1 monclient: found mon.noname-b 2024-03-25T19:36:03.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.962+0000 7f6d0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:03.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.962+0000 7f6d0e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:36:03.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.962+0000 7f6d0e7fc640 1 -- 172.21.15.44:0/1014040821 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d100b44c0 con 0x7f6d101430c0 2024-03-25T19:36:03.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.962+0000 7f6d0e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:36:03.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.962+0000 7f6d0e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:36:03.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.962+0000 7f6d0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:03.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.962+0000 7f6d0e7fc640 1 -- 172.21.15.44:0/1014040821 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6d08001a10 con 0x7f6d101430c0 2024-03-25T19:36:03.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.962+0000 7f6d0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:03.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.962+0000 7f6d0e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:03.965 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.962+0000 7f6d0e7fc640 10 monclient: dump: 2024-03-25T19:36:03.965 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:03.965 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:03.965 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:03.965 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:03.965 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:03.965 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:03.965 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:03.965 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:03.965 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:03.965 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:03.965 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.962+0000 7f6d158f1640 5 monclient: authenticate success, global_id 4208 2024-03-25T19:36:03.965 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.962+0000 7f6d158f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:03.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.964+0000 7f6d0e7fc640 1 -- 172.21.15.44:0/1014040821 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d08001b70 con 0x7f6d101430c0 2024-03-25T19:36:03.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.964+0000 7f6d0e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:03.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.964+0000 7f6d0e7fc640 1 -- 172.21.15.44:0/1014040821 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6d08001f80 con 0x7f6d101430c0 2024-03-25T19:36:03.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.964+0000 7f6d0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:03.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.964+0000 7f6d0e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:03.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.964+0000 7f6d0e7fc640 10 monclient: dump: 2024-03-25T19:36:03.970 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:03.970 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:03.970 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:03.970 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:03.970 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:03.970 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:03.971 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:03.971 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:03.971 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:03.971 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:03.971 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.964+0000 7f6d158f1640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:03.971 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.964+0000 7f6d158f1640 4 set_mon_vals no callback set 2024-03-25T19:36:03.971 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.965+0000 7f6d158f1640 10 monclient: shutdown 2024-03-25T19:36:03.971 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.965+0000 7f6d158f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:03.971 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.965+0000 7f6d158f1640 1 -- 172.21.15.44:0/1014040821 >> v1:172.21.15.44:6790/0 conn(0x7f6d101430c0 legacy=0x7f6d1014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:03.971 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.965+0000 7f6d158f1640 1 -- 172.21.15.44:0/1014040821 shutdown_connections 2024-03-25T19:36:03.971 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.965+0000 7f6d158f1640 1 -- 172.21.15.44:0/1014040821 >> 172.21.15.44:0/1014040821 conn(0x7f6d100b7ea0 msgr2=0x7f6d100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:03.971 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.965+0000 7f6d158f1640 1 -- 172.21.15.44:0/1014040821 shutdown_connections 2024-03-25T19:36:03.971 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.965+0000 7f6d158f1640 1 -- 172.21.15.44:0/1014040821 wait complete. 2024-03-25T19:36:03.971 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.966+0000 7f6d158f1640 10 monclient: build_initial_monmap 2024-03-25T19:36:03.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.966+0000 7f6d158f1640 10 monclient: monmap: 2024-03-25T19:36:03.972 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:36:03.972 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:03.972 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:36:03.972 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:36:03.972 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:03.972 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:36:03.972 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:36:03.972 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:36:03.972 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:36:03.972 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:03.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.966+0000 7f6d158f1640 1 librados: starting msgr at 2024-03-25T19:36:03.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.966+0000 7f6d158f1640 1 librados: starting objecter 2024-03-25T19:36:03.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.967+0000 7f6d158f1640 1 Processor -- start 2024-03-25T19:36:03.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.967+0000 7f6d158f1640 1 -- start start 2024-03-25T19:36:03.973 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.967+0000 7f6d158f1640 1 librados: setting wanted keys 2024-03-25T19:36:03.973 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.967+0000 7f6d158f1640 1 librados: calling monclient init 2024-03-25T19:36:03.973 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.967+0000 7f6d158f1640 10 monclient: init 2024-03-25T19:36:03.973 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.967+0000 7f6d158f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:03.973 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.967+0000 7f6d158f1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:03.973 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.967+0000 7f6d158f1640 10 monclient(hunting): picked mon.noname-a con 0x7f6d100a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:36:03.973 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.967+0000 7f6d158f1640 10 monclient(hunting): picked mon.noname-c con 0x7f6d1013f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:36:03.973 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.967+0000 7f6d158f1640 10 monclient(hunting): picked mon.noname-b con 0x7f6d101430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:36:03.973 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.967+0000 7f6d158f1640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d101daca0 con 0x7f6d100a98f0 2024-03-25T19:36:03.973 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.967+0000 7f6d158f1640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d10138ab0 con 0x7f6d101430c0 2024-03-25T19:36:03.973 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.967+0000 7f6d158f1640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d100aba90 con 0x7f6d1013f710 2024-03-25T19:36:03.973 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.967+0000 7f6d158f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:03.973 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.967+0000 7f6d158f1640 10 monclient(hunting): authenticate will time out at 5428.138751s 2024-03-25T19:36:03.973 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.968+0000 7f6d07fff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f6d1013f710 0x7f6d101db130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:32878/0 (socket says 172.21.15.44:32878) 2024-03-25T19:36:03.973 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.968+0000 7f6d07fff640 1 -- 172.21.15.44:0/3730155657 learned_addr learned my addr 172.21.15.44:0/3730155657 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:36:03.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.968+0000 7f6d0cff9640 10 client.?.objecter ms_handle_connect 0x7f6d1013f710 2024-03-25T19:36:03.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.968+0000 7f6d0cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:03.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.969+0000 7f6d0cff9640 10 client.?.objecter ms_handle_connect 0x7f6d101430c0 2024-03-25T19:36:03.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.969+0000 7f6d0cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:03.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.969+0000 7f6d0cff9640 10 client.?.objecter ms_handle_connect 0x7f6d100a98f0 2024-03-25T19:36:03.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.969+0000 7f6d0cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:03.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.971+0000 7f6d0cff9640 1 -- 172.21.15.44:0/3730155657 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794368386 0 0) 0x7f6d100aba90 con 0x7f6d1013f710 2024-03-25T19:36:03.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.971+0000 7f6d0cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:03.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.971+0000 7f6d0cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:03.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.971+0000 7f6d0cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:03.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.971+0000 7f6d0cff9640 10 monclient(hunting): my global_id is 4211 2024-03-25T19:36:03.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.971+0000 7f6d0cff9640 1 -- 172.21.15.44:0/3730155657 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cf0004eb0 con 0x7f6d1013f710 2024-03-25T19:36:03.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.972+0000 7f6d0cff9640 1 -- 172.21.15.44:0/3730155657 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697502137 0 0) 0x7f6d10138ab0 con 0x7f6d101430c0 2024-03-25T19:36:03.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.972+0000 7f6d0cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:03.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.972+0000 7f6d0cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:03.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.972+0000 7f6d0cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:03.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.972+0000 7f6d0cff9640 10 monclient(hunting): my global_id is 4225 2024-03-25T19:36:03.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.972+0000 7f6d0cff9640 1 -- 172.21.15.44:0/3730155657 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d100aba90 con 0x7f6d101430c0 2024-03-25T19:36:03.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.972+0000 7f6d0cff9640 1 -- 172.21.15.44:0/3730155657 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3163767182 0 0) 0x7f6d101daca0 con 0x7f6d100a98f0 2024-03-25T19:36:03.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.972+0000 7f6d0cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:03.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.972+0000 7f6d0cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:03.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.972+0000 7f6d0cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:03.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.972+0000 7f6d0cff9640 10 monclient(hunting): my global_id is 4239 2024-03-25T19:36:03.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.972+0000 7f6d0cff9640 1 -- 172.21.15.44:0/3730155657 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d10138ab0 con 0x7f6d100a98f0 2024-03-25T19:36:03.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.977+0000 7f6d0cff9640 1 -- 172.21.15.44:0/3730155657 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 539457886 0 0) 0x7f6cf0004eb0 con 0x7f6d1013f710 2024-03-25T19:36:03.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.978+0000 7f6d0cff9640 1 -- 172.21.15.44:0/3730155657 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d101daca0 con 0x7f6d1013f710 2024-03-25T19:36:03.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.978+0000 7f6d0cff9640 1 -- 172.21.15.44:0/3730155657 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6cf8001a30 con 0x7f6d1013f710 2024-03-25T19:36:03.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.978+0000 7f6d0cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:03.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.978+0000 7f6d0cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:03.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.978+0000 7f6d0cff9640 10 monclient(hunting): dump: 2024-03-25T19:36:03.978 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:03.978 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:03.978 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:03.978 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:03.979 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:03.979 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:03.979 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:03.979 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:03.979 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:03.979 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:03.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.978+0000 7f6d0cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:03.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.979+0000 7f6d0cff9640 1 -- 172.21.15.44:0/3730155657 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3322556261 0 0) 0x7f6d10138ab0 con 0x7f6d100a98f0 2024-03-25T19:36:03.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.979+0000 7f6d0cff9640 1 -- 172.21.15.44:0/3730155657 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cf0004eb0 con 0x7f6d100a98f0 2024-03-25T19:36:03.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.979+0000 7f6d0cff9640 1 -- 172.21.15.44:0/3730155657 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6d000017e0 con 0x7f6d100a98f0 2024-03-25T19:36:03.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.979+0000 7f6d0cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:03.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.979+0000 7f6d0cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:03.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.979+0000 7f6d0cff9640 10 monclient(hunting): dump: 2024-03-25T19:36:03.981 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:03.981 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:03.981 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:03.981 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:03.981 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:03.981 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:03.981 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:03.981 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:03.981 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:03.981 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:03.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.979+0000 7f6d0cff9640 1 -- 172.21.15.44:0/3730155657 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3896932607 0 0) 0x7f6d100aba90 con 0x7f6d101430c0 2024-03-25T19:36:03.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.979+0000 7f6d0cff9640 1 -- 172.21.15.44:0/3730155657 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d10138ab0 con 0x7f6d101430c0 2024-03-25T19:36:03.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.979+0000 7f6d0cff9640 1 -- 172.21.15.44:0/3730155657 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6d08001e80 con 0x7f6d101430c0 2024-03-25T19:36:03.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.979+0000 7f6d0cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:03.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.979+0000 7f6d0cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:03.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.979+0000 7f6d0cff9640 10 monclient(hunting): dump: 2024-03-25T19:36:03.982 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:03.982 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:03.982 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:03.982 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:03.982 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:03.982 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:03.982 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:03.982 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:03.982 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:03.982 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:03.987 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.985+0000 7f6d0cff9640 1 -- 172.21.15.44:0/3730155657 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3385089832 0 0) 0x7f6d101daca0 con 0x7f6d1013f710 2024-03-25T19:36:03.987 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.985+0000 7f6d0cff9640 1 -- 172.21.15.44:0/3730155657 >> v1:172.21.15.111:6789/0 conn(0x7f6d101430c0 legacy=0x7f6d101d47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:03.987 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.985+0000 7f6d0cff9640 1 -- 172.21.15.44:0/3730155657 >> v1:172.21.15.44:6789/0 conn(0x7f6d100a98f0 legacy=0x7f6d101383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:03.987 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.985+0000 7f6d0cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:36:03.987 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.985+0000 7f6d0cff9640 1 monclient: found mon.c 2024-03-25T19:36:03.987 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.985+0000 7f6d0cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:03.987 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.985+0000 7f6d0cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:36:03.987 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.985+0000 7f6d0cff9640 1 -- 172.21.15.44:0/3730155657 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d101d41e0 con 0x7f6d1013f710 2024-03-25T19:36:03.987 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.985+0000 7f6d0cff9640 10 monclient: _finish_auth 0 2024-03-25T19:36:03.987 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.985+0000 7f6d0cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:36:03.987 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.985+0000 7f6d0cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.985+0000 7f6d158f1640 5 monclient: authenticate success, global_id 4211 2024-03-25T19:36:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.985+0000 7f6d158f1640 10 monclient: _renew_subs 2024-03-25T19:36:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.985+0000 7f6d158f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:36:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.985+0000 7f6d158f1640 1 -- 172.21.15.44:0/3730155657 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d101db840 con 0x7f6d1013f710 2024-03-25T19:36:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.985+0000 7f6d158f1640 10 client.4211.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.985+0000 7f6d158f1640 10 monclient: _renew_subs 2024-03-25T19:36:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.985+0000 7f6d158f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:36:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.985+0000 7f6d158f1640 1 -- 172.21.15.44:0/3730155657 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d10138f00 con 0x7f6d1013f710 2024-03-25T19:36:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.985+0000 7f6d158f1640 1 librados: init done 2024-03-25T19:36:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.986+0000 7f6d158f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.986+0000 7f6d158f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.987+0000 7f6d158f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:36:03.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:03.987+0000 7f6d158f1640 1 -- 172.21.15.44:0/3730155657 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d100627b0 con 0x7f6d1013f710 2024-03-25T19:36:04.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.009+0000 7f6d0cff9640 1 -- 172.21.15.44:0/3730155657 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6cf8001c70 con 0x7f6d1013f710 2024-03-25T19:36:04.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.009+0000 7f6d0cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:04.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.009+0000 7f6d0cff9640 1 -- 172.21.15.44:0/3730155657 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6cf8003a60 con 0x7f6d1013f710 2024-03-25T19:36:04.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.009+0000 7f6d0cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:04.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.009+0000 7f6d0cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:04.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.009+0000 7f6d0cff9640 10 monclient: dump: 2024-03-25T19:36:04.010 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:04.010 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:04.010 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:04.010 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:04.010 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:04.010 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:04.010 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:04.010 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:04.010 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:04.010 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:04.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.009+0000 7f6d0dffb640 4 set_mon_vals no callback set 2024-03-25T19:36:04.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.010+0000 7f6d0cff9640 1 -- 172.21.15.44:0/3730155657 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f6cf8003c80 con 0x7f6d1013f710 2024-03-25T19:36:04.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.011+0000 7f6d0effd640 10 monclient: get_auth_request con 0x7f6cf006df60 auth_method 0 2024-03-25T19:36:04.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.011+0000 7f6d0cff9640 10 client.4211.objecter ms_handle_connect 0x7f6cf006df60 2024-03-25T19:36:04.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.013+0000 7f6d0cff9640 1 -- 172.21.15.44:0/3730155657 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f6cf80548a0 con 0x7f6d1013f710 2024-03-25T19:36:04.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.013+0000 7f6d0cff9640 10 client.4211.objecter ms_dispatch 0x7f6d10000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:04.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.013+0000 7f6d0cff9640 3 client.4211.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:04.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.013+0000 7f6d0cff9640 3 client.4211.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:04.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.013+0000 7f6d0cff9640 20 client.4211.objecter dump_active .. 0 homeless 2024-03-25T19:36:04.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.296+0000 7f6d0cff9640 1 -- 172.21.15.44:0/3730155657 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6cf80503b0 con 0x7f6d1013f710 2024-03-25T19:36:04.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.296+0000 7f6d0cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:04.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.296+0000 7f6d0cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:04.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.559+0000 7f6d158f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:04.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.559+0000 7f6d158f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:04.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.559+0000 7f6d158f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:36:04.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.559+0000 7f6d158f1640 1 -- 172.21.15.44:0/3730155657 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d101d4370 con 0x7f6d1013f710 2024-03-25T19:36:04.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.607+0000 7f6d0cff9640 1 -- 172.21.15.44:0/3730155657 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f6cf80522e0 con 0x7f6d1013f710 2024-03-25T19:36:04.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.607+0000 7f6d0cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:04.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.607+0000 7f6d0cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:04.608 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:36:04.608 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:04.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.610+0000 7f6d158f1640 10 librados: watch_flush enter 2024-03-25T19:36:04.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.610+0000 7f6d158f1640 10 librados: watch_flush exit 2024-03-25T19:36:04.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.610+0000 7f6d158f1640 20 client.4211.objecter shutdown clearing up homeless session... 2024-03-25T19:36:04.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.610+0000 7f6d158f1640 10 client.4211.objecter successfully canceled tick 2024-03-25T19:36:04.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.610+0000 7f6d158f1640 1 -- 172.21.15.44:0/3730155657 >> v1:172.21.15.111:6800/3883215793 conn(0x7f6cf006df60 legacy=0x7f6cf0070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:04.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.610+0000 7f6d158f1640 10 monclient: shutdown 2024-03-25T19:36:04.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.610+0000 7f6d158f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:04.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.610+0000 7f6d158f1640 1 -- 172.21.15.44:0/3730155657 >> v1:172.21.15.44:6790/0 conn(0x7f6d1013f710 legacy=0x7f6d101db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:04.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.610+0000 7f6d158f1640 1 -- 172.21.15.44:0/3730155657 shutdown_connections 2024-03-25T19:36:04.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.610+0000 7f6d158f1640 1 -- 172.21.15.44:0/3730155657 >> 172.21.15.44:0/3730155657 conn(0x7f6d100b7ea0 msgr2=0x7f6d10074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:04.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.610+0000 7f6d158f1640 1 -- 172.21.15.44:0/3730155657 shutdown_connections 2024-03-25T19:36:04.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.610+0000 7f6d158f1640 1 -- 172.21.15.44:0/3730155657 wait complete. 2024-03-25T19:36:04.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:04.610+0000 7f6d158f1640 1 librados: shutdown 2024-03-25T19:36:04.621 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:06.116 INFO:tasks.ceph.osd.4.smithi111.stderr:2024-03-25T19:36:06.114+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T19:36:06.266 INFO:tasks.ceph.osd.7.smithi111.stderr:2024-03-25T19:36:06.264+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T19:36:06.702 INFO:tasks.ceph.osd.5.smithi111.stderr:2024-03-25T19:36:06.699+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T19:36:07.046 INFO:tasks.ceph.osd.6.smithi111.stderr:2024-03-25T19:36:07.044+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T19:36:08.364 INFO:tasks.ceph.osd.0.smithi044.stderr:2024-03-25T19:36:08.363+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T19:36:08.373 INFO:tasks.ceph.osd.1.smithi044.stderr:2024-03-25T19:36:08.370+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T19:36:08.857 INFO:tasks.ceph.osd.3.smithi044.stderr:2024-03-25T19:36:08.854+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T19:36:09.055 INFO:tasks.ceph.osd.2.smithi044.stderr:2024-03-25T19:36:09.051+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T19:36:11.427 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:11.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.542+0000 7f95f4554640 10 monclient: get_monmap_and_config 2024-03-25T19:36:11.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.542+0000 7f95f4554640 10 monclient: build_initial_monmap 2024-03-25T19:36:11.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.542+0000 7f95f4554640 10 monclient: monmap: 2024-03-25T19:36:11.543 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:36:11.543 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:11.543 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:36:11.543726+0000 2024-03-25T19:36:11.543 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:36:11.543726+0000 2024-03-25T19:36:11.543 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:11.543 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:11.543 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:36:11.543 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:36:11.543 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:36:11.543 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:11.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.543+0000 7f95f4554640 1 Processor -- start 2024-03-25T19:36:11.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.543+0000 7f95f4554640 1 -- start start 2024-03-25T19:36:11.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.543+0000 7f95f4554640 10 monclient: init 2024-03-25T19:36:11.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.544+0000 7f95f4554640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:11.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.544+0000 7f95f4554640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:36:11.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.544+0000 7f95f4554640 10 monclient(hunting): picked mon.noname-b con 0x7f95ec0b4f40 addr v1:172.21.15.44:6790/0 2024-03-25T19:36:11.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.544+0000 7f95f4554640 10 monclient(hunting): picked mon.noname-c con 0x7f95ec13fd90 addr v1:172.21.15.111:6789/0 2024-03-25T19:36:11.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.544+0000 7f95f4554640 10 monclient(hunting): picked mon.noname-a con 0x7f95ec0b4540 addr v1:172.21.15.44:6789/0 2024-03-25T19:36:11.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.544+0000 7f95f4554640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec0aa0c0 con 0x7f95ec0b4540 2024-03-25T19:36:11.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.544+0000 7f95f4554640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec0ab170 con 0x7f95ec13fd90 2024-03-25T19:36:11.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.544+0000 7f95f4554640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec0a9ba0 con 0x7f95ec0b4f40 2024-03-25T19:36:11.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.544+0000 7f95f4554640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:11.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.544+0000 7f95f4554640 10 monclient(hunting): authenticate will time out at 5435.715214s 2024-03-25T19:36:11.549 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.549+0000 7f95f1aef640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f95ec13fd90 0x7f95ec1423f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:39138/0 (socket says 172.21.15.44:39138) 2024-03-25T19:36:11.549 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.549+0000 7f95f1aef640 1 -- 172.21.15.44:0/1906666464 learned_addr learned my addr 172.21.15.44:0/1906666464 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:36:11.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.563+0000 7f95f12ee640 1 -- 172.21.15.44:0/1906666464 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2629170850 0 0) 0x7f95ec0ab170 con 0x7f95ec13fd90 2024-03-25T19:36:11.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.563+0000 7f95f12ee640 10 monclient(hunting): _negotiate 2024-03-25T19:36:11.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.563+0000 7f95f12ee640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:11.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.563+0000 7f95f12ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:11.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.563+0000 7f95f12ee640 10 monclient(hunting): my global_id is 4252 2024-03-25T19:36:11.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.563+0000 7f95f12ee640 1 -- 172.21.15.44:0/1906666464 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95d8003520 con 0x7f95ec13fd90 2024-03-25T19:36:11.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.564+0000 7f95f12ee640 1 -- 172.21.15.44:0/1906666464 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559154229 0 0) 0x7f95ec0aa0c0 con 0x7f95ec0b4540 2024-03-25T19:36:11.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.564+0000 7f95f12ee640 10 monclient(hunting): _negotiate 2024-03-25T19:36:11.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.564+0000 7f95f12ee640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:11.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.564+0000 7f95f12ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:11.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.564+0000 7f95f12ee640 10 monclient(hunting): my global_id is 4266 2024-03-25T19:36:11.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.564+0000 7f95f12ee640 1 -- 172.21.15.44:0/1906666464 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95ec0ab170 con 0x7f95ec0b4540 2024-03-25T19:36:11.567 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.567+0000 7f95f12ee640 1 -- 172.21.15.44:0/1906666464 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826775331 0 0) 0x7f95ec0a9ba0 con 0x7f95ec0b4f40 2024-03-25T19:36:11.567 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.568+0000 7f95f12ee640 10 monclient(hunting): _negotiate 2024-03-25T19:36:11.567 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.568+0000 7f95f12ee640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:11.567 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.568+0000 7f95f12ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:11.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.568+0000 7f95f12ee640 10 monclient(hunting): my global_id is 4238 2024-03-25T19:36:11.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.568+0000 7f95f12ee640 1 -- 172.21.15.44:0/1906666464 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95ec0aa0c0 con 0x7f95ec0b4f40 2024-03-25T19:36:11.577 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.577+0000 7f95f12ee640 1 -- 172.21.15.44:0/1906666464 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3271300423 0 0) 0x7f95d8003520 con 0x7f95ec13fd90 2024-03-25T19:36:11.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.577+0000 7f95f12ee640 1 -- 172.21.15.44:0/1906666464 >> v1:172.21.15.44:6790/0 conn(0x7f95ec0b4f40 legacy=0x7f95ec13f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:11.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.577+0000 7f95f12ee640 1 -- 172.21.15.44:0/1906666464 >> v1:172.21.15.44:6789/0 conn(0x7f95ec0b4540 legacy=0x7f95ec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:11.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.577+0000 7f95f12ee640 10 monclient: _finish_hunting 0 2024-03-25T19:36:11.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.577+0000 7f95f12ee640 1 monclient: found mon.noname-c 2024-03-25T19:36:11.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.577+0000 7f95f12ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:11.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.577+0000 7f95f12ee640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:36:11.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.577+0000 7f95f12ee640 1 -- 172.21.15.44:0/1906666464 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ec0a9680 con 0x7f95ec13fd90 2024-03-25T19:36:11.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.577+0000 7f95f12ee640 10 monclient: _finish_auth 0 2024-03-25T19:36:11.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.577+0000 7f95f12ee640 10 monclient: _check_auth_tickets 2024-03-25T19:36:11.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.577+0000 7f95f12ee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:11.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.577+0000 7f95f12ee640 1 -- 172.21.15.44:0/1906666464 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f95dc001690 con 0x7f95ec13fd90 2024-03-25T19:36:11.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.577+0000 7f95f12ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:11.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.577+0000 7f95f12ee640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:11.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.577+0000 7f95f12ee640 10 monclient: dump: 2024-03-25T19:36:11.579 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:11.579 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:11.579 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:11.579 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:11.579 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:11.579 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:11.579 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:11.579 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:11.579 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:11.579 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:11.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.577+0000 7f95f4554640 5 monclient: authenticate success, global_id 4252 2024-03-25T19:36:11.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.577+0000 7f95f4554640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:11.581 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.581+0000 7f95f12ee640 1 -- 172.21.15.44:0/1906666464 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95dc0017f0 con 0x7f95ec13fd90 2024-03-25T19:36:11.581 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.581+0000 7f95f12ee640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:11.581 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.581+0000 7f95f4554640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:11.581 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.581+0000 7f95f4554640 4 set_mon_vals no callback set 2024-03-25T19:36:11.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.582+0000 7f95f12ee640 1 -- 172.21.15.44:0/1906666464 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f95dc001c00 con 0x7f95ec13fd90 2024-03-25T19:36:11.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.582+0000 7f95f12ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:11.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.582+0000 7f95f12ee640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:11.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.582+0000 7f95f12ee640 10 monclient: dump: 2024-03-25T19:36:11.583 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:11.583 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:11.583 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:11.583 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:11.583 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:11.583 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:11.583 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:11.583 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:11.583 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:11.583 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:11.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.582+0000 7f95f4554640 10 monclient: shutdown 2024-03-25T19:36:11.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.582+0000 7f95f4554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:11.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.582+0000 7f95f4554640 1 -- 172.21.15.44:0/1906666464 >> v1:172.21.15.111:6789/0 conn(0x7f95ec13fd90 legacy=0x7f95ec1423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:11.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.582+0000 7f95f4554640 1 -- 172.21.15.44:0/1906666464 shutdown_connections 2024-03-25T19:36:11.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.582+0000 7f95f4554640 1 -- 172.21.15.44:0/1906666464 >> 172.21.15.44:0/1906666464 conn(0x7f95ec0617a0 msgr2=0x7f95ec061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:11.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.582+0000 7f95f4554640 1 -- 172.21.15.44:0/1906666464 shutdown_connections 2024-03-25T19:36:11.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.582+0000 7f95f4554640 1 -- 172.21.15.44:0/1906666464 wait complete. 2024-03-25T19:36:11.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.583+0000 7f95f4554640 10 monclient: build_initial_monmap 2024-03-25T19:36:11.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.583+0000 7f95f4554640 10 monclient: monmap: 2024-03-25T19:36:11.585 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:36:11.585 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:11.585 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:36:11.585 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:36:11.585 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:11.585 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:36:11.586 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:36:11.586 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:36:11.586 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:36:11.586 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:11.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.583+0000 7f95f4554640 1 librados: starting msgr at 2024-03-25T19:36:11.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.583+0000 7f95f4554640 1 librados: starting objecter 2024-03-25T19:36:11.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.583+0000 7f95f4554640 1 Processor -- start 2024-03-25T19:36:11.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.583+0000 7f95f4554640 1 -- start start 2024-03-25T19:36:11.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.583+0000 7f95f4554640 1 librados: setting wanted keys 2024-03-25T19:36:11.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.583+0000 7f95f4554640 1 librados: calling monclient init 2024-03-25T19:36:11.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.583+0000 7f95f4554640 10 monclient: init 2024-03-25T19:36:11.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.584+0000 7f95f4554640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:11.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.584+0000 7f95f4554640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:36:11.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.584+0000 7f95f4554640 10 monclient(hunting): picked mon.noname-b con 0x7f95ec0b4540 addr v1:172.21.15.111:6789/0 2024-03-25T19:36:11.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.584+0000 7f95f4554640 10 monclient(hunting): picked mon.noname-c con 0x7f95ec0b4f40 addr v1:172.21.15.44:6790/0 2024-03-25T19:36:11.587 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.584+0000 7f95f4554640 10 monclient(hunting): picked mon.noname-a con 0x7f95ec13fd90 addr v1:172.21.15.44:6789/0 2024-03-25T19:36:11.587 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.584+0000 7f95f4554640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec1db850 con 0x7f95ec13fd90 2024-03-25T19:36:11.587 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.584+0000 7f95f4554640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec0b05f0 con 0x7f95ec0b4540 2024-03-25T19:36:11.587 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.584+0000 7f95f4554640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec1d4ff0 con 0x7f95ec0b4f40 2024-03-25T19:36:11.587 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.584+0000 7f95f4554640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:11.587 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.584+0000 7f95f4554640 10 monclient(hunting): authenticate will time out at 5435.755734s 2024-03-25T19:36:11.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.595+0000 7f95f22f0640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f95ec0b4540 0x7f95ec1daa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:39168/0 (socket says 172.21.15.44:39168) 2024-03-25T19:36:11.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.595+0000 7f95f22f0640 1 -- 172.21.15.44:0/2647778362 learned_addr learned my addr 172.21.15.44:0/2647778362 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:36:11.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.596+0000 7f95e2ffd640 10 client.?.objecter ms_handle_connect 0x7f95ec0b4540 2024-03-25T19:36:11.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.596+0000 7f95e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:11.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.600+0000 7f95e2ffd640 1 -- 172.21.15.44:0/2647778362 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 953193193 0 0) 0x7f95ec0b05f0 con 0x7f95ec0b4540 2024-03-25T19:36:11.600 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.600+0000 7f95e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:11.600 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.600+0000 7f95e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:11.600 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.600+0000 7f95e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:11.600 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.600+0000 7f95e2ffd640 10 monclient(hunting): my global_id is 4255 2024-03-25T19:36:11.600 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.600+0000 7f95e2ffd640 1 -- 172.21.15.44:0/2647778362 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95d0004eb0 con 0x7f95ec0b4540 2024-03-25T19:36:11.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.603+0000 7f95e2ffd640 10 client.?.objecter ms_handle_connect 0x7f95ec13fd90 2024-03-25T19:36:11.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.603+0000 7f95e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:11.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.606+0000 7f95e2ffd640 10 client.?.objecter ms_handle_connect 0x7f95ec0b4f40 2024-03-25T19:36:11.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.606+0000 7f95e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:11.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.613+0000 7f95e2ffd640 1 -- 172.21.15.44:0/2647778362 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3405744347 0 0) 0x7f95d0004eb0 con 0x7f95ec0b4540 2024-03-25T19:36:11.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.613+0000 7f95e2ffd640 1 -- 172.21.15.44:0/2647778362 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95ec0b05f0 con 0x7f95ec0b4540 2024-03-25T19:36:11.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.613+0000 7f95e2ffd640 1 -- 172.21.15.44:0/2647778362 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f95e8001a10 con 0x7f95ec0b4540 2024-03-25T19:36:11.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.613+0000 7f95e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:11.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.613+0000 7f95e2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:11.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.613+0000 7f95e2ffd640 10 monclient(hunting): dump: 2024-03-25T19:36:11.614 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:11.614 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:11.614 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:11.614 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:11.614 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:11.614 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:11.614 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:11.614 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:11.614 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:11.614 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:11.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.613+0000 7f95e2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:11.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.619+0000 7f95e2ffd640 1 -- 172.21.15.44:0/2647778362 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679384154 0 0) 0x7f95ec1d4ff0 con 0x7f95ec0b4f40 2024-03-25T19:36:11.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.619+0000 7f95e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:11.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.619+0000 7f95e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:11.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.619+0000 7f95e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:11.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.619+0000 7f95e2ffd640 10 monclient(hunting): my global_id is 4241 2024-03-25T19:36:11.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.619+0000 7f95e2ffd640 1 -- 172.21.15.44:0/2647778362 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95d0004eb0 con 0x7f95ec0b4f40 2024-03-25T19:36:11.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.622+0000 7f95e2ffd640 1 -- 172.21.15.44:0/2647778362 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177602859 0 0) 0x7f95ec1db850 con 0x7f95ec13fd90 2024-03-25T19:36:11.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.622+0000 7f95e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:11.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.622+0000 7f95e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:11.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.622+0000 7f95e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:11.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.622+0000 7f95e2ffd640 10 monclient(hunting): my global_id is 4269 2024-03-25T19:36:11.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.622+0000 7f95e2ffd640 1 -- 172.21.15.44:0/2647778362 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95ec1d4ff0 con 0x7f95ec13fd90 2024-03-25T19:36:11.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.624+0000 7f95e2ffd640 1 -- 172.21.15.44:0/2647778362 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3541781557 0 0) 0x7f95ec0b05f0 con 0x7f95ec0b4540 2024-03-25T19:36:11.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.624+0000 7f95e2ffd640 1 -- 172.21.15.44:0/2647778362 >> v1:172.21.15.44:6790/0 conn(0x7f95ec0b4f40 legacy=0x7f95ec0aef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:11.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.624+0000 7f95e2ffd640 1 -- 172.21.15.44:0/2647778362 >> v1:172.21.15.44:6789/0 conn(0x7f95ec13fd90 legacy=0x7f95ec1db140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:11.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.624+0000 7f95e2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:36:11.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.624+0000 7f95e2ffd640 1 monclient: found mon.b 2024-03-25T19:36:11.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.624+0000 7f95e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:11.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.624+0000 7f95e2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:11.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.624+0000 7f95e2ffd640 1 -- 172.21.15.44:0/2647778362 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ec1d8130 con 0x7f95ec0b4540 2024-03-25T19:36:11.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.624+0000 7f95e2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:36:11.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.624+0000 7f95e2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:36:11.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.624+0000 7f95e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:11.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.624+0000 7f95f4554640 5 monclient: authenticate success, global_id 4255 2024-03-25T19:36:11.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.624+0000 7f95f4554640 10 monclient: _renew_subs 2024-03-25T19:36:11.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.624+0000 7f95f4554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:11.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.624+0000 7f95f4554640 1 -- 172.21.15.44:0/2647778362 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95ec1d8380 con 0x7f95ec0b4540 2024-03-25T19:36:11.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.624+0000 7f95f4554640 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:11.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.624+0000 7f95f4554640 10 monclient: _renew_subs 2024-03-25T19:36:11.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.624+0000 7f95f4554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:11.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.624+0000 7f95f4554640 1 -- 172.21.15.44:0/2647778362 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95ec0b0f40 con 0x7f95ec0b4540 2024-03-25T19:36:11.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.624+0000 7f95f4554640 1 librados: init done 2024-03-25T19:36:11.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.625+0000 7f95f4554640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:11.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.625+0000 7f95f4554640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:11.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.625+0000 7f95f4554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:11.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.625+0000 7f95f4554640 1 -- 172.21.15.44:0/2647778362 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95ec0627b0 con 0x7f95ec0b4540 2024-03-25T19:36:11.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.850+0000 7f95e2ffd640 1 -- 172.21.15.44:0/2647778362 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95e8001cb0 con 0x7f95ec0b4540 2024-03-25T19:36:11.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.850+0000 7f95e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:11.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.850+0000 7f95e2ffd640 1 -- 172.21.15.44:0/2647778362 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f95e8003a60 con 0x7f95ec0b4540 2024-03-25T19:36:11.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.850+0000 7f95e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:11.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.850+0000 7f95e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:11.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.850+0000 7f95e2ffd640 10 monclient: dump: 2024-03-25T19:36:11.851 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:11.851 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:11.851 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:11.851 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:11.851 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:11.851 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:11.851 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:11.851 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:11.852 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:11.852 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:11.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.850+0000 7f95f0aed640 4 set_mon_vals no callback set 2024-03-25T19:36:11.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.899+0000 7f95e2ffd640 1 -- 172.21.15.44:0/2647778362 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f95e8003bc0 con 0x7f95ec0b4540 2024-03-25T19:36:11.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.900+0000 7f95f1aef640 10 monclient: get_auth_request con 0x7f95d006de80 auth_method 0 2024-03-25T19:36:11.900 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.900+0000 7f95e2ffd640 10 client.4255.objecter ms_handle_connect 0x7f95d006de80 2024-03-25T19:36:11.901 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.901+0000 7f95e2ffd640 1 -- 172.21.15.44:0/2647778362 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f95e8054890 con 0x7f95ec0b4540 2024-03-25T19:36:11.901 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.901+0000 7f95e2ffd640 10 client.4255.objecter ms_dispatch 0x7f95ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:11.901 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.901+0000 7f95e2ffd640 3 client.4255.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:11.901 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.901+0000 7f95e2ffd640 3 client.4255.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:11.902 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.901+0000 7f95e2ffd640 20 client.4255.objecter dump_active .. 0 homeless 2024-03-25T19:36:11.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.904+0000 7f95e2ffd640 1 -- 172.21.15.44:0/2647778362 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f95e80501d0 con 0x7f95ec0b4540 2024-03-25T19:36:11.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.904+0000 7f95e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:11.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:11.904+0000 7f95e2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:12.172 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:12.172+0000 7f95f4554640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:12.172 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:12.172+0000 7f95f4554640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:12.172 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:12.172+0000 7f95f4554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:12.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:12.172+0000 7f95f4554640 1 -- 172.21.15.44:0/2647778362 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95ec13a480 con 0x7f95ec0b4540 2024-03-25T19:36:12.194 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:12.194+0000 7f95e2ffd640 1 -- 172.21.15.44:0/2647778362 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f95e80522d0 con 0x7f95ec0b4540 2024-03-25T19:36:12.194 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:12.194+0000 7f95e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:12.194 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:12.194+0000 7f95e2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:12.194 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:36:12.194 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:12.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:12.196+0000 7f95f4554640 10 librados: watch_flush enter 2024-03-25T19:36:12.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:12.196+0000 7f95f4554640 10 librados: watch_flush exit 2024-03-25T19:36:12.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:12.196+0000 7f95f4554640 20 client.4255.objecter shutdown clearing up homeless session... 2024-03-25T19:36:12.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:12.196+0000 7f95f4554640 10 client.4255.objecter successfully canceled tick 2024-03-25T19:36:12.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:12.196+0000 7f95f4554640 1 -- 172.21.15.44:0/2647778362 >> v1:172.21.15.111:6800/3883215793 conn(0x7f95d006de80 legacy=0x7f95d0070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:12.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:12.196+0000 7f95f4554640 10 monclient: shutdown 2024-03-25T19:36:12.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:12.196+0000 7f95f4554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:12.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:12.196+0000 7f95f4554640 1 -- 172.21.15.44:0/2647778362 >> v1:172.21.15.111:6789/0 conn(0x7f95ec0b4540 legacy=0x7f95ec1daa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:12.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:12.196+0000 7f95f4554640 1 -- 172.21.15.44:0/2647778362 shutdown_connections 2024-03-25T19:36:12.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:12.196+0000 7f95f4554640 1 -- 172.21.15.44:0/2647778362 >> 172.21.15.44:0/2647778362 conn(0x7f95ec0617a0 msgr2=0x7f95ec061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:12.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:12.196+0000 7f95f4554640 1 -- 172.21.15.44:0/2647778362 shutdown_connections 2024-03-25T19:36:12.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:12.196+0000 7f95f4554640 1 -- 172.21.15.44:0/2647778362 wait complete. 2024-03-25T19:36:12.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:12.196+0000 7f95f4554640 1 librados: shutdown 2024-03-25T19:36:12.209 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:19.016 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:19.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.131+0000 7ff29e46b640 10 monclient: get_monmap_and_config 2024-03-25T19:36:19.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.131+0000 7ff29e46b640 10 monclient: build_initial_monmap 2024-03-25T19:36:19.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.131+0000 7ff29e46b640 10 monclient: monmap: 2024-03-25T19:36:19.131 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:36:19.131 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:19.131 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:36:19.132319+0000 2024-03-25T19:36:19.131 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:36:19.132319+0000 2024-03-25T19:36:19.131 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:19.131 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:19.131 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:36:19.131 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:36:19.131 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:36:19.132 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:19.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.132+0000 7ff29e46b640 1 Processor -- start 2024-03-25T19:36:19.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.132+0000 7ff29e46b640 1 -- start start 2024-03-25T19:36:19.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.132+0000 7ff29e46b640 10 monclient: init 2024-03-25T19:36:19.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.132+0000 7ff29e46b640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:19.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.132+0000 7ff29e46b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:36:19.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.132+0000 7ff29e46b640 10 monclient(hunting): picked mon.noname-b con 0x7ff2980b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T19:36:19.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.132+0000 7ff29e46b640 10 monclient(hunting): picked mon.noname-a con 0x7ff2980b1730 addr v1:172.21.15.44:6789/0 2024-03-25T19:36:19.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.132+0000 7ff29e46b640 10 monclient(hunting): picked mon.noname-c con 0x7ff29814be90 addr v1:172.21.15.111:6789/0 2024-03-25T19:36:19.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.132+0000 7ff29e46b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2980af940 con 0x7ff2980b1730 2024-03-25T19:36:19.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.132+0000 7ff29e46b640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2980b0a70 con 0x7ff29814be90 2024-03-25T19:36:19.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.133+0000 7ff29e46b640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2980b0d50 con 0x7ff2980b4d60 2024-03-25T19:36:19.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.133+0000 7ff29e46b640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:19.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.133+0000 7ff29e46b640 10 monclient(hunting): authenticate will time out at 5443.303888s 2024-03-25T19:36:19.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.146+0000 7ff2977fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7ff2980b1730 0x7ff2980b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57190/0 (socket says 172.21.15.44:57190) 2024-03-25T19:36:19.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.146+0000 7ff2977fe640 1 -- 172.21.15.44:0/2558628038 learned_addr learned my addr 172.21.15.44:0/2558628038 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:36:19.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.156+0000 7ff296ffd640 1 -- 172.21.15.44:0/2558628038 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653885720 0 0) 0x7ff2980af940 con 0x7ff2980b1730 2024-03-25T19:36:19.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.156+0000 7ff296ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:19.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.156+0000 7ff296ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:19.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.156+0000 7ff296ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:19.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.156+0000 7ff296ffd640 10 monclient(hunting): my global_id is 4272 2024-03-25T19:36:19.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.156+0000 7ff296ffd640 1 -- 172.21.15.44:0/2558628038 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff280003520 con 0x7ff2980b1730 2024-03-25T19:36:19.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.164+0000 7ff296ffd640 1 -- 172.21.15.44:0/2558628038 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 923336557 0 0) 0x7ff280003520 con 0x7ff2980b1730 2024-03-25T19:36:19.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.164+0000 7ff296ffd640 1 -- 172.21.15.44:0/2558628038 >> v1:172.21.15.44:6790/0 conn(0x7ff2980b4d60 legacy=0x7ff29813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:19.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.164+0000 7ff296ffd640 1 -- 172.21.15.44:0/2558628038 >> v1:172.21.15.111:6789/0 conn(0x7ff29814be90 legacy=0x7ff29814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:19.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.164+0000 7ff296ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:36:19.164 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.164+0000 7ff296ffd640 1 monclient: found mon.noname-a 2024-03-25T19:36:19.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.164+0000 7ff296ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:19.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.164+0000 7ff296ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:36:19.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.164+0000 7ff296ffd640 1 -- 172.21.15.44:0/2558628038 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2980b1030 con 0x7ff2980b1730 2024-03-25T19:36:19.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.164+0000 7ff296ffd640 10 monclient: _finish_auth 0 2024-03-25T19:36:19.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.164+0000 7ff296ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:36:19.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.164+0000 7ff296ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:19.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.164+0000 7ff296ffd640 1 -- 172.21.15.44:0/2558628038 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff28c001690 con 0x7ff2980b1730 2024-03-25T19:36:19.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.164+0000 7ff296ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:19.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.164+0000 7ff296ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:19.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.164+0000 7ff296ffd640 10 monclient: dump: 2024-03-25T19:36:19.165 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:19.165 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:19.165 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:19.165 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:19.166 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:19.166 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:19.166 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:19.166 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:19.166 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:19.166 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:19.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.164+0000 7ff29e46b640 5 monclient: authenticate success, global_id 4272 2024-03-25T19:36:19.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.164+0000 7ff29e46b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:19.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.182+0000 7ff296ffd640 1 -- 172.21.15.44:0/2558628038 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff28c0017f0 con 0x7ff2980b1730 2024-03-25T19:36:19.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.182+0000 7ff296ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:19.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.182+0000 7ff296ffd640 1 -- 172.21.15.44:0/2558628038 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff28c001c00 con 0x7ff2980b1730 2024-03-25T19:36:19.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.182+0000 7ff29e46b640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:19.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.182+0000 7ff29e46b640 4 set_mon_vals no callback set 2024-03-25T19:36:19.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.182+0000 7ff296ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:19.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.182+0000 7ff296ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:19.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.182+0000 7ff296ffd640 10 monclient: dump: 2024-03-25T19:36:19.183 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:19.183 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:19.183 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:19.183 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:19.183 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:19.184 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:19.184 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:19.184 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:19.184 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:19.184 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:19.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.183+0000 7ff29e46b640 10 monclient: shutdown 2024-03-25T19:36:19.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.183+0000 7ff29e46b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:19.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.183+0000 7ff29e46b640 1 -- 172.21.15.44:0/2558628038 >> v1:172.21.15.44:6789/0 conn(0x7ff2980b1730 legacy=0x7ff2980b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:19.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.183+0000 7ff29e46b640 1 -- 172.21.15.44:0/2558628038 shutdown_connections 2024-03-25T19:36:19.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.183+0000 7ff29e46b640 1 -- 172.21.15.44:0/2558628038 >> 172.21.15.44:0/2558628038 conn(0x7ff2980617a0 msgr2=0x7ff298061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:19.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.183+0000 7ff29e46b640 1 -- 172.21.15.44:0/2558628038 shutdown_connections 2024-03-25T19:36:19.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.183+0000 7ff29e46b640 1 -- 172.21.15.44:0/2558628038 wait complete. 2024-03-25T19:36:19.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.184+0000 7ff29e46b640 10 monclient: build_initial_monmap 2024-03-25T19:36:19.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.184+0000 7ff29e46b640 10 monclient: monmap: 2024-03-25T19:36:19.186 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:36:19.186 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:19.186 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:36:19.186 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:36:19.186 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:19.186 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:36:19.186 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:36:19.186 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:36:19.186 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:36:19.186 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:19.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.185+0000 7ff29e46b640 1 librados: starting msgr at 2024-03-25T19:36:19.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.185+0000 7ff29e46b640 1 librados: starting objecter 2024-03-25T19:36:19.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.185+0000 7ff29e46b640 1 Processor -- start 2024-03-25T19:36:19.187 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.185+0000 7ff29e46b640 1 -- start start 2024-03-25T19:36:19.187 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.185+0000 7ff29e46b640 1 librados: setting wanted keys 2024-03-25T19:36:19.187 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.185+0000 7ff29e46b640 1 librados: calling monclient init 2024-03-25T19:36:19.187 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.185+0000 7ff29e46b640 10 monclient: init 2024-03-25T19:36:19.187 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.186+0000 7ff29e46b640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:19.188 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.186+0000 7ff29e46b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:36:19.188 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.186+0000 7ff29e46b640 10 monclient(hunting): picked mon.noname-c con 0x7ff2980b1730 addr v1:172.21.15.44:6790/0 2024-03-25T19:36:19.188 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.186+0000 7ff29e46b640 10 monclient(hunting): picked mon.noname-a con 0x7ff2980b4d60 addr v1:172.21.15.44:6789/0 2024-03-25T19:36:19.188 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.186+0000 7ff29e46b640 10 monclient(hunting): picked mon.noname-b con 0x7ff29814be90 addr v1:172.21.15.111:6789/0 2024-03-25T19:36:19.188 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.186+0000 7ff29e46b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2981420d0 con 0x7ff2980b4d60 2024-03-25T19:36:19.188 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.186+0000 7ff29e46b640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff298138270 con 0x7ff29814be90 2024-03-25T19:36:19.188 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.186+0000 7ff29e46b640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff298138400 con 0x7ff2980b1730 2024-03-25T19:36:19.188 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.186+0000 7ff29e46b640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:19.188 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.186+0000 7ff29e46b640 10 monclient(hunting): authenticate will time out at 5443.357139s 2024-03-25T19:36:19.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.198+0000 7ff29ca08640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff29814be90 0x7ff298134d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:41632/0 (socket says 172.21.15.44:41632) 2024-03-25T19:36:19.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.198+0000 7ff29ca08640 1 -- 172.21.15.44:0/1062527986 learned_addr learned my addr 172.21.15.44:0/1062527986 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:36:19.199 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.199+0000 7ff294ff9640 10 client.?.objecter ms_handle_connect 0x7ff29814be90 2024-03-25T19:36:19.199 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.199+0000 7ff294ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:19.204 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.205+0000 7ff294ff9640 10 client.?.objecter ms_handle_connect 0x7ff2980b4d60 2024-03-25T19:36:19.204 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.205+0000 7ff294ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:19.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.208+0000 7ff294ff9640 1 -- 172.21.15.44:0/1062527986 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165877313 0 0) 0x7ff298138270 con 0x7ff29814be90 2024-03-25T19:36:19.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.208+0000 7ff294ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:19.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.208+0000 7ff294ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:19.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.208+0000 7ff294ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:19.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.208+0000 7ff294ff9640 10 monclient(hunting): my global_id is 4261 2024-03-25T19:36:19.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.208+0000 7ff294ff9640 1 -- 172.21.15.44:0/1062527986 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff284004eb0 con 0x7ff29814be90 2024-03-25T19:36:19.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.211+0000 7ff294ff9640 10 client.?.objecter ms_handle_connect 0x7ff2980b1730 2024-03-25T19:36:19.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.211+0000 7ff294ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:19.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.215+0000 7ff294ff9640 1 -- 172.21.15.44:0/1062527986 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 907978660 0 0) 0x7ff2981420d0 con 0x7ff2980b4d60 2024-03-25T19:36:19.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.215+0000 7ff294ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:19.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.215+0000 7ff294ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:19.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.215+0000 7ff294ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:19.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.215+0000 7ff294ff9640 10 monclient(hunting): my global_id is 4275 2024-03-25T19:36:19.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.215+0000 7ff294ff9640 1 -- 172.21.15.44:0/1062527986 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff298138270 con 0x7ff2980b4d60 2024-03-25T19:36:19.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.215+0000 7ff294ff9640 1 -- 172.21.15.44:0/1062527986 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 835911620 0 0) 0x7ff284004eb0 con 0x7ff29814be90 2024-03-25T19:36:19.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.215+0000 7ff294ff9640 1 -- 172.21.15.44:0/1062527986 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2981420d0 con 0x7ff29814be90 2024-03-25T19:36:19.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.215+0000 7ff294ff9640 1 -- 172.21.15.44:0/1062527986 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff290001c30 con 0x7ff29814be90 2024-03-25T19:36:19.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.215+0000 7ff294ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:19.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.215+0000 7ff294ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:19.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.215+0000 7ff294ff9640 10 monclient(hunting): dump: 2024-03-25T19:36:19.216 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:19.216 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:19.216 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:19.216 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:19.216 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:19.216 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:19.216 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:19.217 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:19.217 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:19.217 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:19.217 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.215+0000 7ff294ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:19.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.227+0000 7ff294ff9640 1 -- 172.21.15.44:0/1062527986 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1527999173 0 0) 0x7ff298138400 con 0x7ff2980b1730 2024-03-25T19:36:19.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.227+0000 7ff294ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:19.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.227+0000 7ff294ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:19.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.227+0000 7ff294ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:19.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.227+0000 7ff294ff9640 10 monclient(hunting): my global_id is 4247 2024-03-25T19:36:19.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.227+0000 7ff294ff9640 1 -- 172.21.15.44:0/1062527986 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff284004eb0 con 0x7ff2980b1730 2024-03-25T19:36:19.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.228+0000 7ff294ff9640 1 -- 172.21.15.44:0/1062527986 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3179774340 0 0) 0x7ff2981420d0 con 0x7ff29814be90 2024-03-25T19:36:19.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.228+0000 7ff294ff9640 1 -- 172.21.15.44:0/1062527986 >> v1:172.21.15.44:6790/0 conn(0x7ff2980b1730 legacy=0x7ff2981db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:19.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.228+0000 7ff294ff9640 1 -- 172.21.15.44:0/1062527986 >> v1:172.21.15.44:6789/0 conn(0x7ff2980b4d60 legacy=0x7ff2981405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:19.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.228+0000 7ff294ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:36:19.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.228+0000 7ff294ff9640 1 monclient: found mon.b 2024-03-25T19:36:19.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.228+0000 7ff294ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:19.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.228+0000 7ff294ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:19.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.228+0000 7ff294ff9640 1 -- 172.21.15.44:0/1062527986 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff298142480 con 0x7ff29814be90 2024-03-25T19:36:19.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.228+0000 7ff294ff9640 10 monclient: _finish_auth 0 2024-03-25T19:36:19.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.228+0000 7ff294ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:36:19.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.228+0000 7ff294ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:19.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.228+0000 7ff29e46b640 5 monclient: authenticate success, global_id 4261 2024-03-25T19:36:19.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.228+0000 7ff29e46b640 10 monclient: _renew_subs 2024-03-25T19:36:19.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.228+0000 7ff29e46b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:19.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.228+0000 7ff29e46b640 1 -- 172.21.15.44:0/1062527986 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2981426d0 con 0x7ff29814be90 2024-03-25T19:36:19.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.228+0000 7ff29e46b640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:19.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.228+0000 7ff29e46b640 10 monclient: _renew_subs 2024-03-25T19:36:19.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.228+0000 7ff29e46b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:19.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.228+0000 7ff29e46b640 1 -- 172.21.15.44:0/1062527986 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2981dbae0 con 0x7ff29814be90 2024-03-25T19:36:19.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.228+0000 7ff29e46b640 1 librados: init done 2024-03-25T19:36:19.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.229+0000 7ff29e46b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:19.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.229+0000 7ff29e46b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:19.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.229+0000 7ff29e46b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:19.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.229+0000 7ff29e46b640 1 -- 172.21.15.44:0/1062527986 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2981401f0 con 0x7ff29814be90 2024-03-25T19:36:19.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.442+0000 7ff294ff9640 1 -- 172.21.15.44:0/1062527986 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff290003990 con 0x7ff29814be90 2024-03-25T19:36:19.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.442+0000 7ff294ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:19.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.442+0000 7ff2967fc640 4 set_mon_vals no callback set 2024-03-25T19:36:19.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.503+0000 7ff294ff9640 1 -- 172.21.15.44:0/1062527986 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff290003d20 con 0x7ff29814be90 2024-03-25T19:36:19.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.503+0000 7ff294ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:19.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.503+0000 7ff294ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:19.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.503+0000 7ff294ff9640 10 monclient: dump: 2024-03-25T19:36:19.504 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:19.504 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:19.504 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:19.504 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:19.504 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:19.504 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:19.504 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:19.504 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:19.504 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:19.504 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:19.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.504+0000 7ff294ff9640 1 -- 172.21.15.44:0/1062527986 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7ff290003f00 con 0x7ff29814be90 2024-03-25T19:36:19.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.505+0000 7ff297fff640 10 monclient: get_auth_request con 0x7ff28406de00 auth_method 0 2024-03-25T19:36:19.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.505+0000 7ff294ff9640 10 client.4261.objecter ms_handle_connect 0x7ff28406de00 2024-03-25T19:36:19.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.507+0000 7ff294ff9640 1 -- 172.21.15.44:0/1062527986 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7ff290054a20 con 0x7ff29814be90 2024-03-25T19:36:19.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.507+0000 7ff294ff9640 10 client.4261.objecter ms_dispatch 0x7ff298000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:19.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.507+0000 7ff294ff9640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:19.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.507+0000 7ff294ff9640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:19.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.507+0000 7ff294ff9640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-25T19:36:19.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.511+0000 7ff294ff9640 1 -- 172.21.15.44:0/1062527986 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff290050530 con 0x7ff29814be90 2024-03-25T19:36:19.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.511+0000 7ff294ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:19.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.511+0000 7ff294ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:19.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.743+0000 7ff29e46b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:19.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.743+0000 7ff29e46b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:19.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.743+0000 7ff29e46b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:19.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.743+0000 7ff29e46b640 1 -- 172.21.15.44:0/1062527986 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2980627b0 con 0x7ff29814be90 2024-03-25T19:36:19.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.773+0000 7ff294ff9640 1 -- 172.21.15.44:0/1062527986 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7ff290052460 con 0x7ff29814be90 2024-03-25T19:36:19.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.773+0000 7ff294ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:19.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.773+0000 7ff294ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:19.774 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:36:19.774 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:19.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.776+0000 7ff29e46b640 10 librados: watch_flush enter 2024-03-25T19:36:19.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.776+0000 7ff29e46b640 10 librados: watch_flush exit 2024-03-25T19:36:19.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.776+0000 7ff29e46b640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-25T19:36:19.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.776+0000 7ff29e46b640 10 client.4261.objecter successfully canceled tick 2024-03-25T19:36:19.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.776+0000 7ff29e46b640 1 -- 172.21.15.44:0/1062527986 >> v1:172.21.15.111:6800/3883215793 conn(0x7ff28406de00 legacy=0x7ff2840702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:19.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.776+0000 7ff29e46b640 10 monclient: shutdown 2024-03-25T19:36:19.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.776+0000 7ff29e46b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:19.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.776+0000 7ff29e46b640 1 -- 172.21.15.44:0/1062527986 >> v1:172.21.15.111:6789/0 conn(0x7ff29814be90 legacy=0x7ff298134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:19.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.776+0000 7ff29e46b640 1 -- 172.21.15.44:0/1062527986 shutdown_connections 2024-03-25T19:36:19.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.776+0000 7ff29e46b640 1 -- 172.21.15.44:0/1062527986 >> 172.21.15.44:0/1062527986 conn(0x7ff2980617a0 msgr2=0x7ff2980b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:19.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.776+0000 7ff29e46b640 1 -- 172.21.15.44:0/1062527986 shutdown_connections 2024-03-25T19:36:19.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.776+0000 7ff29e46b640 1 -- 172.21.15.44:0/1062527986 wait complete. 2024-03-25T19:36:19.778 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:19.776+0000 7ff29e46b640 1 librados: shutdown 2024-03-25T19:36:19.787 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:23.653 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:01:14.555 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:23.723 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:01:14.625 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:23.891 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:01:14.771 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:23.928 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:01:14.830 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:23.952 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:01:14.854 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:23.955 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:01:14.835 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:23.995 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:01:14.897 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:24.177 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:01:15.056 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:24.246 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:01:15.148 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:26.594 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:26.710 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.710+0000 7f32b23e2640 10 monclient: get_monmap_and_config 2024-03-25T19:36:26.710 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.710+0000 7f32b23e2640 10 monclient: build_initial_monmap 2024-03-25T19:36:26.710 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.710+0000 7f32b23e2640 10 monclient: monmap: 2024-03-25T19:36:26.711 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:36:26.711 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:26.711 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:36:26.711958+0000 2024-03-25T19:36:26.711 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:36:26.711958+0000 2024-03-25T19:36:26.711 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:26.711 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:26.711 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:36:26.711 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:36:26.711 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:36:26.711 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:26.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.711+0000 7f32b23e2640 1 Processor -- start 2024-03-25T19:36:26.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.711+0000 7f32b23e2640 1 -- start start 2024-03-25T19:36:26.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.711+0000 7f32b23e2640 10 monclient: init 2024-03-25T19:36:26.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.712+0000 7f32b23e2640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:26.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.712+0000 7f32b23e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:36:26.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.712+0000 7f32b23e2640 10 monclient(hunting): picked mon.noname-c con 0x7f32ac0a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:36:26.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.712+0000 7f32b23e2640 10 monclient(hunting): picked mon.noname-b con 0x7f32ac13f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:36:26.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.712+0000 7f32b23e2640 10 monclient(hunting): picked mon.noname-a con 0x7f32ac1430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:36:26.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.712+0000 7f32b23e2640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32ac142520 con 0x7f32ac1430c0 2024-03-25T19:36:26.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.712+0000 7f32b23e2640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32ac0b28a0 con 0x7f32ac0a98f0 2024-03-25T19:36:26.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.712+0000 7f32b23e2640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32ac0b2a30 con 0x7f32ac13f710 2024-03-25T19:36:26.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.712+0000 7f32b23e2640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:26.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.712+0000 7f32b23e2640 10 monclient(hunting): authenticate will time out at 5450.883558s 2024-03-25T19:36:26.717 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.717+0000 7f32abfff640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f32ac0a98f0 0x7f32ac0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:45758/0 (socket says 172.21.15.44:45758) 2024-03-25T19:36:26.717 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.717+0000 7f32abfff640 1 -- 172.21.15.44:0/1949898529 learned_addr learned my addr 172.21.15.44:0/1949898529 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:36:26.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.727+0000 7f32aaffd640 1 -- 172.21.15.44:0/1949898529 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703117054 0 0) 0x7f32ac0b28a0 con 0x7f32ac0a98f0 2024-03-25T19:36:26.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.727+0000 7f32aaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:26.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.727+0000 7f32aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:26.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.727+0000 7f32aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:26.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.727+0000 7f32aaffd640 10 monclient(hunting): my global_id is 4264 2024-03-25T19:36:26.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.727+0000 7f32aaffd640 1 -- 172.21.15.44:0/1949898529 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3294003520 con 0x7f32ac0a98f0 2024-03-25T19:36:26.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.729+0000 7f32aaffd640 1 -- 172.21.15.44:0/1949898529 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430967263 0 0) 0x7f32ac142520 con 0x7f32ac1430c0 2024-03-25T19:36:26.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.729+0000 7f32aaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:26.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.729+0000 7f32aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:26.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.729+0000 7f32aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:26.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.729+0000 7f32aaffd640 10 monclient(hunting): my global_id is 4278 2024-03-25T19:36:26.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.729+0000 7f32aaffd640 1 -- 172.21.15.44:0/1949898529 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32ac0b28a0 con 0x7f32ac1430c0 2024-03-25T19:36:26.734 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.735+0000 7f32aaffd640 1 -- 172.21.15.44:0/1949898529 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1526367947 0 0) 0x7f32ac0b2a30 con 0x7f32ac13f710 2024-03-25T19:36:26.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.735+0000 7f32aaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:26.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.735+0000 7f32aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:26.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.735+0000 7f32aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:26.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.735+0000 7f32aaffd640 10 monclient(hunting): my global_id is 4250 2024-03-25T19:36:26.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.735+0000 7f32aaffd640 1 -- 172.21.15.44:0/1949898529 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32ac142520 con 0x7f32ac13f710 2024-03-25T19:36:26.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.751+0000 7f32aaffd640 1 -- 172.21.15.44:0/1949898529 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1453444574 0 0) 0x7f32ac142520 con 0x7f32ac13f710 2024-03-25T19:36:26.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.751+0000 7f32aaffd640 1 -- 172.21.15.44:0/1949898529 >> v1:172.21.15.111:6789/0 conn(0x7f32ac0a98f0 legacy=0x7f32ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:26.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.751+0000 7f32aaffd640 1 -- 172.21.15.44:0/1949898529 >> v1:172.21.15.44:6789/0 conn(0x7f32ac1430c0 legacy=0x7f32ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:26.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.751+0000 7f32aaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:36:26.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.751+0000 7f32aaffd640 1 monclient: found mon.noname-b 2024-03-25T19:36:26.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.751+0000 7f32aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:26.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.751+0000 7f32aaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:36:26.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.751+0000 7f32aaffd640 1 -- 172.21.15.44:0/1949898529 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32ac0b44c0 con 0x7f32ac13f710 2024-03-25T19:36:26.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.751+0000 7f32aaffd640 10 monclient: _finish_auth 0 2024-03-25T19:36:26.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.751+0000 7f32aaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:36:26.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.751+0000 7f32aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:26.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.751+0000 7f32aaffd640 1 -- 172.21.15.44:0/1949898529 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f32a00015b0 con 0x7f32ac13f710 2024-03-25T19:36:26.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.751+0000 7f32aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:26.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.751+0000 7f32aaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:26.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.751+0000 7f32aaffd640 10 monclient: dump: 2024-03-25T19:36:26.752 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:26.752 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:26.752 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:26.752 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:26.752 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:26.752 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:26.753 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:26.753 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:26.753 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:26.753 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:26.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.751+0000 7f32b23e2640 5 monclient: authenticate success, global_id 4250 2024-03-25T19:36:26.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.751+0000 7f32b23e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:26.762 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.762+0000 7f32aaffd640 1 -- 172.21.15.44:0/1949898529 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32a0001710 con 0x7f32ac13f710 2024-03-25T19:36:26.762 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.762+0000 7f32aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:26.762 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.762+0000 7f32aaffd640 1 -- 172.21.15.44:0/1949898529 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f32a0001b60 con 0x7f32ac13f710 2024-03-25T19:36:26.762 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.762+0000 7f32b23e2640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:26.762 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.762+0000 7f32b23e2640 4 set_mon_vals no callback set 2024-03-25T19:36:26.762 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.762+0000 7f32aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:26.763 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.762+0000 7f32aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:26.763 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.762+0000 7f32aaffd640 10 monclient: dump: 2024-03-25T19:36:26.763 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:26.763 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:26.763 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:26.763 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:26.763 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:26.763 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:26.763 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:26.763 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:26.763 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:26.763 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:26.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.763+0000 7f32b23e2640 10 monclient: shutdown 2024-03-25T19:36:26.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.763+0000 7f32b23e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:26.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.763+0000 7f32b23e2640 1 -- 172.21.15.44:0/1949898529 >> v1:172.21.15.44:6790/0 conn(0x7f32ac13f710 legacy=0x7f32ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:26.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.763+0000 7f32b23e2640 1 -- 172.21.15.44:0/1949898529 shutdown_connections 2024-03-25T19:36:26.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.763+0000 7f32b23e2640 1 -- 172.21.15.44:0/1949898529 >> 172.21.15.44:0/1949898529 conn(0x7f32ac0b7ea0 msgr2=0x7f32ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:26.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.763+0000 7f32b23e2640 1 -- 172.21.15.44:0/1949898529 shutdown_connections 2024-03-25T19:36:26.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.763+0000 7f32b23e2640 1 -- 172.21.15.44:0/1949898529 wait complete. 2024-03-25T19:36:26.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.764+0000 7f32b23e2640 10 monclient: build_initial_monmap 2024-03-25T19:36:26.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.764+0000 7f32b23e2640 10 monclient: monmap: 2024-03-25T19:36:26.765 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:36:26.765 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:26.765 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:36:26.765 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:36:26.765 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:26.766 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:36:26.766 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:36:26.766 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:36:26.766 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:36:26.766 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:26.766 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.764+0000 7f32b23e2640 1 librados: starting msgr at 2024-03-25T19:36:26.766 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.764+0000 7f32b23e2640 1 librados: starting objecter 2024-03-25T19:36:26.766 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.764+0000 7f32b23e2640 1 Processor -- start 2024-03-25T19:36:26.766 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.765+0000 7f32b23e2640 1 -- start start 2024-03-25T19:36:26.766 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.765+0000 7f32b23e2640 1 librados: setting wanted keys 2024-03-25T19:36:26.766 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.765+0000 7f32b23e2640 1 librados: calling monclient init 2024-03-25T19:36:26.766 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.765+0000 7f32b23e2640 10 monclient: init 2024-03-25T19:36:26.767 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.765+0000 7f32b23e2640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:26.767 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.765+0000 7f32b23e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:36:26.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.765+0000 7f32b23e2640 10 monclient(hunting): picked mon.noname-b con 0x7f32ac0a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:36:26.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.766+0000 7f32b23e2640 10 monclient(hunting): picked mon.noname-c con 0x7f32ac13f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:36:26.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.766+0000 7f32b23e2640 10 monclient(hunting): picked mon.noname-a con 0x7f32ac1430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:36:26.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.766+0000 7f32b23e2640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32ac1391d0 con 0x7f32ac1430c0 2024-03-25T19:36:26.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.766+0000 7f32b23e2640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32ac1377f0 con 0x7f32ac0a98f0 2024-03-25T19:36:26.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.766+0000 7f32b23e2640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32ac136cc0 con 0x7f32ac13f710 2024-03-25T19:36:26.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.766+0000 7f32b23e2640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:26.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.766+0000 7f32b23e2640 10 monclient(hunting): authenticate will time out at 5450.936950s 2024-03-25T19:36:26.774 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.774+0000 7f32abfff640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f32ac0a98f0 0x7f32ac1370e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:45784/0 (socket says 172.21.15.44:45784) 2024-03-25T19:36:26.774 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.774+0000 7f32abfff640 1 -- 172.21.15.44:0/2020798857 learned_addr learned my addr 172.21.15.44:0/2020798857 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:36:26.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.779+0000 7f32a8ff9640 10 client.?.objecter ms_handle_connect 0x7f32ac0a98f0 2024-03-25T19:36:26.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.779+0000 7f32a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:26.784 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.784+0000 7f32a8ff9640 10 client.?.objecter ms_handle_connect 0x7f32ac13f710 2024-03-25T19:36:26.784 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.784+0000 7f32a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:26.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.790+0000 7f32a8ff9640 10 client.?.objecter ms_handle_connect 0x7f32ac1430c0 2024-03-25T19:36:26.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.790+0000 7f32a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:26.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.791+0000 7f32a8ff9640 1 -- 172.21.15.44:0/2020798857 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972063776 0 0) 0x7f32ac1377f0 con 0x7f32ac0a98f0 2024-03-25T19:36:26.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.791+0000 7f32a8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:26.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.791+0000 7f32a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:26.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.791+0000 7f32a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:26.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.791+0000 7f32a8ff9640 10 monclient(hunting): my global_id is 4267 2024-03-25T19:36:26.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.791+0000 7f32a8ff9640 1 -- 172.21.15.44:0/2020798857 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3298004eb0 con 0x7f32ac0a98f0 2024-03-25T19:36:26.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.800+0000 7f32a8ff9640 1 -- 172.21.15.44:0/2020798857 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250893862 0 0) 0x7f32ac136cc0 con 0x7f32ac13f710 2024-03-25T19:36:26.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.800+0000 7f32a8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:26.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.800+0000 7f32a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:26.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.800+0000 7f32a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:26.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.800+0000 7f32a8ff9640 10 monclient(hunting): my global_id is 4253 2024-03-25T19:36:26.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.800+0000 7f32a8ff9640 1 -- 172.21.15.44:0/2020798857 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32ac1377f0 con 0x7f32ac13f710 2024-03-25T19:36:26.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.801+0000 7f32a8ff9640 1 -- 172.21.15.44:0/2020798857 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1023030447 0 0) 0x7f3298004eb0 con 0x7f32ac0a98f0 2024-03-25T19:36:26.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.801+0000 7f32a8ff9640 1 -- 172.21.15.44:0/2020798857 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32ac136cc0 con 0x7f32ac0a98f0 2024-03-25T19:36:26.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.801+0000 7f32a8ff9640 1 -- 172.21.15.44:0/2020798857 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f329c001950 con 0x7f32ac0a98f0 2024-03-25T19:36:26.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.801+0000 7f32a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:26.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.801+0000 7f32a8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:26.802 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.801+0000 7f32a8ff9640 10 monclient(hunting): dump: 2024-03-25T19:36:26.802 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:26.802 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:26.802 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:26.802 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:26.802 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:26.802 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:26.802 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:26.802 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:26.802 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:26.802 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:26.802 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.801+0000 7f32a8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:26.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.808+0000 7f32a8ff9640 1 -- 172.21.15.44:0/2020798857 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3189662569 0 0) 0x7f32ac1391d0 con 0x7f32ac1430c0 2024-03-25T19:36:26.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.808+0000 7f32a8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:26.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.808+0000 7f32a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:26.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.808+0000 7f32a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:26.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.808+0000 7f32a8ff9640 10 monclient(hunting): my global_id is 4281 2024-03-25T19:36:26.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.808+0000 7f32a8ff9640 1 -- 172.21.15.44:0/2020798857 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3298004eb0 con 0x7f32ac1430c0 2024-03-25T19:36:26.815 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.815+0000 7f32a8ff9640 1 -- 172.21.15.44:0/2020798857 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3839608094 0 0) 0x7f3298004eb0 con 0x7f32ac1430c0 2024-03-25T19:36:26.815 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.815+0000 7f32a8ff9640 1 -- 172.21.15.44:0/2020798857 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32ac1391d0 con 0x7f32ac1430c0 2024-03-25T19:36:26.815 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.815+0000 7f32a8ff9640 1 -- 172.21.15.44:0/2020798857 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f32a4001bd0 con 0x7f32ac1430c0 2024-03-25T19:36:26.815 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.815+0000 7f32a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:26.815 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.815+0000 7f32a8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:26.815 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.815+0000 7f32a8ff9640 10 monclient(hunting): dump: 2024-03-25T19:36:26.815 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:26.815 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:26.815 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:26.816 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:26.816 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:26.816 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:26.816 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:26.816 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:26.816 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:26.816 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:26.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.818+0000 7f32a8ff9640 1 -- 172.21.15.44:0/2020798857 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2092436636 0 0) 0x7f32ac136cc0 con 0x7f32ac0a98f0 2024-03-25T19:36:26.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.818+0000 7f32a8ff9640 1 -- 172.21.15.44:0/2020798857 >> v1:172.21.15.44:6790/0 conn(0x7f32ac13f710 legacy=0x7f32ac1365b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:26.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.818+0000 7f32a8ff9640 1 -- 172.21.15.44:0/2020798857 >> v1:172.21.15.44:6789/0 conn(0x7f32ac1430c0 legacy=0x7f32ac132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:26.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.818+0000 7f32a8ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:36:26.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.818+0000 7f32a8ff9640 1 monclient: found mon.b 2024-03-25T19:36:26.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.818+0000 7f32a8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:26.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.818+0000 7f32a8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:26.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.818+0000 7f32a8ff9640 1 -- 172.21.15.44:0/2020798857 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32ac1396f0 con 0x7f32ac0a98f0 2024-03-25T19:36:26.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.818+0000 7f32a8ff9640 10 monclient: _finish_auth 0 2024-03-25T19:36:26.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.818+0000 7f32a8ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:36:26.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.818+0000 7f32a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:26.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.818+0000 7f32b23e2640 5 monclient: authenticate success, global_id 4267 2024-03-25T19:36:26.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.818+0000 7f32b23e2640 10 monclient: _renew_subs 2024-03-25T19:36:26.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.818+0000 7f32b23e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:26.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.818+0000 7f32b23e2640 1 -- 172.21.15.44:0/2020798857 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32ac1399c0 con 0x7f32ac0a98f0 2024-03-25T19:36:26.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.818+0000 7f32b23e2640 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:26.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.818+0000 7f32b23e2640 10 monclient: _renew_subs 2024-03-25T19:36:26.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.818+0000 7f32b23e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:26.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.818+0000 7f32b23e2640 1 -- 172.21.15.44:0/2020798857 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32ac13b360 con 0x7f32ac0a98f0 2024-03-25T19:36:26.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.818+0000 7f32b23e2640 1 librados: init done 2024-03-25T19:36:26.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.819+0000 7f32b23e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:26.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.819+0000 7f32b23e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:26.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.819+0000 7f32b23e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:26.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:26.819+0000 7f32b23e2640 1 -- 172.21.15.44:0/2020798857 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32ac0b3ef0 con 0x7f32ac0a98f0 2024-03-25T19:36:27.090 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.090+0000 7f32a8ff9640 1 -- 172.21.15.44:0/2020798857 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f329c0047e0 con 0x7f32ac0a98f0 2024-03-25T19:36:27.090 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.090+0000 7f32a8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:27.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.090+0000 7f32a8ff9640 1 -- 172.21.15.44:0/2020798857 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f329c004af0 con 0x7f32ac0a98f0 2024-03-25T19:36:27.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.090+0000 7f32a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:27.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.090+0000 7f32a8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:27.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.090+0000 7f32a8ff9640 10 monclient: dump: 2024-03-25T19:36:27.091 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:27.093 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:27.094 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:27.094 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:27.094 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:27.094 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:27.094 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:27.094 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:27.094 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:27.094 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:27.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.090+0000 7f32aa7fc640 4 set_mon_vals no callback set 2024-03-25T19:36:27.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.091+0000 7f32a8ff9640 1 -- 172.21.15.44:0/2020798857 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f329c01aac0 con 0x7f32ac0a98f0 2024-03-25T19:36:27.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.092+0000 7f32ab7fe640 10 monclient: get_auth_request con 0x7f329806df40 auth_method 0 2024-03-25T19:36:27.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.092+0000 7f32a8ff9640 10 client.4267.objecter ms_handle_connect 0x7f329806df40 2024-03-25T19:36:27.100 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.101+0000 7f32a8ff9640 1 -- 172.21.15.44:0/2020798857 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f329c0548e0 con 0x7f32ac0a98f0 2024-03-25T19:36:27.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.101+0000 7f32a8ff9640 10 client.4267.objecter ms_dispatch 0x7f32ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:27.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.101+0000 7f32a8ff9640 3 client.4267.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:27.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.101+0000 7f32a8ff9640 3 client.4267.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:27.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.101+0000 7f32a8ff9640 20 client.4267.objecter dump_active .. 0 homeless 2024-03-25T19:36:27.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.101+0000 7f32a8ff9640 1 -- 172.21.15.44:0/2020798857 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f329c050220 con 0x7f32ac0a98f0 2024-03-25T19:36:27.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.102+0000 7f32a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:27.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.102+0000 7f32a8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:27.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.361+0000 7f32b23e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:27.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.361+0000 7f32b23e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:27.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.361+0000 7f32b23e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:27.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.361+0000 7f32b23e2640 1 -- 172.21.15.44:0/2020798857 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32ac139030 con 0x7f32ac0a98f0 2024-03-25T19:36:27.478 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.478+0000 7f32a8ff9640 1 -- 172.21.15.44:0/2020798857 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f32ac139030 con 0x7f32ac0a98f0 2024-03-25T19:36:27.478 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.478+0000 7f32a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:27.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.478+0000 7f32a8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:27.479 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:36:27.479 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:27.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.480+0000 7f32b23e2640 10 librados: watch_flush enter 2024-03-25T19:36:27.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.480+0000 7f32b23e2640 10 librados: watch_flush exit 2024-03-25T19:36:27.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.480+0000 7f32b23e2640 20 client.4267.objecter shutdown clearing up homeless session... 2024-03-25T19:36:27.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.480+0000 7f32b23e2640 10 client.4267.objecter successfully canceled tick 2024-03-25T19:36:27.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.480+0000 7f32b23e2640 1 -- 172.21.15.44:0/2020798857 >> v1:172.21.15.111:6800/3883215793 conn(0x7f329806df40 legacy=0x7f3298070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:27.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.480+0000 7f32b23e2640 10 monclient: shutdown 2024-03-25T19:36:27.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.480+0000 7f32b23e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:27.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.480+0000 7f32b23e2640 1 -- 172.21.15.44:0/2020798857 >> v1:172.21.15.111:6789/0 conn(0x7f32ac0a98f0 legacy=0x7f32ac1370e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:27.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.480+0000 7f32b23e2640 1 -- 172.21.15.44:0/2020798857 shutdown_connections 2024-03-25T19:36:27.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.480+0000 7f32b23e2640 1 -- 172.21.15.44:0/2020798857 >> 172.21.15.44:0/2020798857 conn(0x7f32ac0b7ea0 msgr2=0x7f32ac13d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:27.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.481+0000 7f32b23e2640 1 -- 172.21.15.44:0/2020798857 shutdown_connections 2024-03-25T19:36:27.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.481+0000 7f32b23e2640 1 -- 172.21.15.44:0/2020798857 wait complete. 2024-03-25T19:36:27.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:27.481+0000 7f32b23e2640 1 librados: shutdown 2024-03-25T19:36:27.492 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:34.298 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:34.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.405+0000 7f75efdd0640 10 monclient: get_monmap_and_config 2024-03-25T19:36:34.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.405+0000 7f75efdd0640 10 monclient: build_initial_monmap 2024-03-25T19:36:34.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.406+0000 7f75efdd0640 10 monclient: monmap: 2024-03-25T19:36:34.406 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:36:34.406 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:34.406 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:36:34.407147+0000 2024-03-25T19:36:34.406 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:36:34.407147+0000 2024-03-25T19:36:34.406 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:34.406 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:34.406 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:36:34.406 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:36:34.407 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:36:34.407 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:34.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.406+0000 7f75efdd0640 1 Processor -- start 2024-03-25T19:36:34.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.406+0000 7f75efdd0640 1 -- start start 2024-03-25T19:36:34.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.406+0000 7f75efdd0640 10 monclient: init 2024-03-25T19:36:34.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.407+0000 7f75efdd0640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:34.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.407+0000 7f75efdd0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:36:34.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.407+0000 7f75efdd0640 10 monclient(hunting): picked mon.noname-a con 0x7f75e80ae6e0 addr v1:172.21.15.44:6789/0 2024-03-25T19:36:34.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.407+0000 7f75efdd0640 10 monclient(hunting): picked mon.noname-c con 0x7f75e806f080 addr v1:172.21.15.111:6789/0 2024-03-25T19:36:34.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.407+0000 7f75efdd0640 10 monclient(hunting): picked mon.noname-b con 0x7f75e80704e0 addr v1:172.21.15.44:6790/0 2024-03-25T19:36:34.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.407+0000 7f75efdd0640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e80718d0 con 0x7f75e80ae6e0 2024-03-25T19:36:34.409 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.407+0000 7f75efdd0640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e806fa30 con 0x7f75e806f080 2024-03-25T19:36:34.409 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.407+0000 7f75efdd0640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e8070090 con 0x7f75e80704e0 2024-03-25T19:36:34.409 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.407+0000 7f75efdd0640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:34.409 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.407+0000 7f75efdd0640 10 monclient(hunting): authenticate will time out at 5458.578250s 2024-03-25T19:36:34.409 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.408+0000 7f75ed36b640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f75e806f080 0x7f75e8071c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:41136/0 (socket says 172.21.15.44:41136) 2024-03-25T19:36:34.409 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.408+0000 7f75ed36b640 1 -- 172.21.15.44:0/673354601 learned_addr learned my addr 172.21.15.44:0/673354601 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:36:34.409 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.408+0000 7f75ee36d640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f75e80704e0 0x7f75e814a910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:42246/0 (socket says 172.21.15.44:42246) 2024-03-25T19:36:34.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.477+0000 7f75ecb6a640 1 -- 172.21.15.44:0/673354601 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1289114285 0 0) 0x7f75e80718d0 con 0x7f75e80ae6e0 2024-03-25T19:36:34.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.477+0000 7f75ecb6a640 10 monclient(hunting): _negotiate 2024-03-25T19:36:34.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.477+0000 7f75ecb6a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:34.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.477+0000 7f75ecb6a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:34.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.477+0000 7f75ecb6a640 10 monclient(hunting): my global_id is 4284 2024-03-25T19:36:34.478 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.477+0000 7f75ecb6a640 1 -- 172.21.15.44:0/673354601 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75d0005730 con 0x7f75e80ae6e0 2024-03-25T19:36:34.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.501+0000 7f75ecb6a640 1 -- 172.21.15.44:0/673354601 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 795951581 0 0) 0x7f75d0005730 con 0x7f75e80ae6e0 2024-03-25T19:36:34.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.501+0000 7f75ecb6a640 1 -- 172.21.15.44:0/673354601 >> v1:172.21.15.44:6790/0 conn(0x7f75e80704e0 legacy=0x7f75e814a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:34.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.501+0000 7f75ecb6a640 1 -- 172.21.15.44:0/673354601 >> v1:172.21.15.111:6789/0 conn(0x7f75e806f080 legacy=0x7f75e8071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:34.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.501+0000 7f75ecb6a640 10 monclient: _finish_hunting 0 2024-03-25T19:36:34.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.501+0000 7f75ecb6a640 1 monclient: found mon.noname-a 2024-03-25T19:36:34.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.501+0000 7f75ecb6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:34.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.501+0000 7f75ecb6a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:36:34.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.501+0000 7f75ecb6a640 1 -- 172.21.15.44:0/673354601 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e806f510 con 0x7f75e80ae6e0 2024-03-25T19:36:34.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.501+0000 7f75ecb6a640 10 monclient: _finish_auth 0 2024-03-25T19:36:34.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.501+0000 7f75ecb6a640 10 monclient: _check_auth_tickets 2024-03-25T19:36:34.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.501+0000 7f75ecb6a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:34.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.501+0000 7f75ecb6a640 1 -- 172.21.15.44:0/673354601 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f75e0001690 con 0x7f75e80ae6e0 2024-03-25T19:36:34.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.501+0000 7f75efdd0640 5 monclient: authenticate success, global_id 4284 2024-03-25T19:36:34.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.501+0000 7f75efdd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:34.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.501+0000 7f75ecb6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:34.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.501+0000 7f75ecb6a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:34.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.501+0000 7f75ecb6a640 10 monclient: dump: 2024-03-25T19:36:34.502 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:34.503 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:34.503 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:34.503 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:34.503 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:34.503 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:34.503 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:34.503 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:34.503 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:34.503 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:34.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.501+0000 7f75efdd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:34.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.513+0000 7f75ecb6a640 1 -- 172.21.15.44:0/673354601 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75e00017f0 con 0x7f75e80ae6e0 2024-03-25T19:36:34.514 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.513+0000 7f75ecb6a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:34.514 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.514+0000 7f75ecb6a640 1 -- 172.21.15.44:0/673354601 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f75e0001c00 con 0x7f75e80ae6e0 2024-03-25T19:36:34.514 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.514+0000 7f75ecb6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:34.514 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.514+0000 7f75ecb6a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:34.514 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.514+0000 7f75ecb6a640 10 monclient: dump: 2024-03-25T19:36:34.514 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:34.514 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:34.514 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:34.514 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:34.514 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:34.514 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:34.514 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:34.515 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:34.515 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:34.515 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:34.515 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.514+0000 7f75efdd0640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:34.515 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.514+0000 7f75efdd0640 4 set_mon_vals no callback set 2024-03-25T19:36:34.517 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.514+0000 7f75efdd0640 10 monclient: shutdown 2024-03-25T19:36:34.517 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.514+0000 7f75efdd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:34.517 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.514+0000 7f75efdd0640 1 -- 172.21.15.44:0/673354601 >> v1:172.21.15.44:6789/0 conn(0x7f75e80ae6e0 legacy=0x7f75e813c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:34.517 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.514+0000 7f75efdd0640 1 -- 172.21.15.44:0/673354601 shutdown_connections 2024-03-25T19:36:34.517 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.514+0000 7f75efdd0640 1 -- 172.21.15.44:0/673354601 >> 172.21.15.44:0/673354601 conn(0x7f75e806b630 msgr2=0x7f75e80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:34.517 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.514+0000 7f75efdd0640 1 -- 172.21.15.44:0/673354601 shutdown_connections 2024-03-25T19:36:34.517 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.514+0000 7f75efdd0640 1 -- 172.21.15.44:0/673354601 wait complete. 2024-03-25T19:36:34.517 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.515+0000 7f75efdd0640 10 monclient: build_initial_monmap 2024-03-25T19:36:34.517 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.515+0000 7f75efdd0640 10 monclient: monmap: 2024-03-25T19:36:34.517 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:36:34.518 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:34.518 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:36:34.518 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:36:34.518 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:34.518 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:36:34.518 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:36:34.518 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:36:34.518 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:36:34.518 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:34.518 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.515+0000 7f75efdd0640 1 librados: starting msgr at 2024-03-25T19:36:34.518 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.515+0000 7f75efdd0640 1 librados: starting objecter 2024-03-25T19:36:34.518 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.516+0000 7f75efdd0640 1 Processor -- start 2024-03-25T19:36:34.518 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.516+0000 7f75efdd0640 1 -- start start 2024-03-25T19:36:34.518 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.516+0000 7f75efdd0640 1 librados: setting wanted keys 2024-03-25T19:36:34.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.516+0000 7f75efdd0640 1 librados: calling monclient init 2024-03-25T19:36:34.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.516+0000 7f75efdd0640 10 monclient: init 2024-03-25T19:36:34.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.516+0000 7f75efdd0640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:34.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.516+0000 7f75efdd0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:36:34.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.516+0000 7f75efdd0640 10 monclient(hunting): picked mon.noname-b con 0x7f75e80704e0 addr v1:172.21.15.111:6789/0 2024-03-25T19:36:34.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.516+0000 7f75efdd0640 10 monclient(hunting): picked mon.noname-a con 0x7f75e80ae6e0 addr v1:172.21.15.44:6789/0 2024-03-25T19:36:34.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.516+0000 7f75efdd0640 10 monclient(hunting): picked mon.noname-c con 0x7f75e8141570 addr v1:172.21.15.44:6790/0 2024-03-25T19:36:34.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.516+0000 7f75efdd0640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e81432c0 con 0x7f75e80ae6e0 2024-03-25T19:36:34.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.516+0000 7f75efdd0640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e8141070 con 0x7f75e80704e0 2024-03-25T19:36:34.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.516+0000 7f75efdd0640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e81420d0 con 0x7f75e8141570 2024-03-25T19:36:34.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.516+0000 7f75efdd0640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:34.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.516+0000 7f75efdd0640 10 monclient(hunting): authenticate will time out at 5458.687519s 2024-03-25T19:36:34.525 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.525+0000 7f75edb6c640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f75e80704e0 0x7f75e81db080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:41178/0 (socket says 172.21.15.44:41178) 2024-03-25T19:36:34.525 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.525+0000 7f75edb6c640 1 -- 172.21.15.44:0/4237623206 learned_addr learned my addr 172.21.15.44:0/4237623206 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:36:34.525 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.525+0000 7f75e67fc640 10 client.?.objecter ms_handle_connect 0x7f75e80704e0 2024-03-25T19:36:34.525 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.525+0000 7f75e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:34.529 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.529+0000 7f75e67fc640 1 -- 172.21.15.44:0/4237623206 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286192213 0 0) 0x7f75e8141070 con 0x7f75e80704e0 2024-03-25T19:36:34.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.529+0000 7f75e67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:34.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.529+0000 7f75e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:34.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.529+0000 7f75e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:34.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.529+0000 7f75e67fc640 10 monclient(hunting): my global_id is 4273 2024-03-25T19:36:34.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.530+0000 7f75e67fc640 1 -- 172.21.15.44:0/4237623206 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75d4004eb0 con 0x7f75e80704e0 2024-03-25T19:36:34.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.538+0000 7f75e67fc640 10 client.?.objecter ms_handle_connect 0x7f75e8141570 2024-03-25T19:36:34.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.539+0000 7f75e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:34.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.544+0000 7f75e67fc640 10 client.?.objecter ms_handle_connect 0x7f75e80ae6e0 2024-03-25T19:36:34.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.544+0000 7f75e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:34.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.546+0000 7f75e67fc640 1 -- 172.21.15.44:0/4237623206 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1652816570 0 0) 0x7f75d4004eb0 con 0x7f75e80704e0 2024-03-25T19:36:34.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.546+0000 7f75e67fc640 1 -- 172.21.15.44:0/4237623206 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75e8141070 con 0x7f75e80704e0 2024-03-25T19:36:34.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.546+0000 7f75e67fc640 1 -- 172.21.15.44:0/4237623206 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f75e0001690 con 0x7f75e80704e0 2024-03-25T19:36:34.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.546+0000 7f75e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:34.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.546+0000 7f75e67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:34.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.546+0000 7f75e67fc640 10 monclient(hunting): dump: 2024-03-25T19:36:34.547 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:34.547 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:34.547 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:34.547 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:34.547 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:34.547 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:34.547 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:34.547 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:34.547 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:34.547 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:34.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.546+0000 7f75e67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:34.550 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.551+0000 7f75e67fc640 1 -- 172.21.15.44:0/4237623206 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2459752171 0 0) 0x7f75e81420d0 con 0x7f75e8141570 2024-03-25T19:36:34.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.551+0000 7f75e67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:34.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.551+0000 7f75e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:34.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.551+0000 7f75e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:34.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.551+0000 7f75e67fc640 10 monclient(hunting): my global_id is 4256 2024-03-25T19:36:34.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.551+0000 7f75e67fc640 1 -- 172.21.15.44:0/4237623206 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75d4004eb0 con 0x7f75e8141570 2024-03-25T19:36:34.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.561+0000 7f75e67fc640 1 -- 172.21.15.44:0/4237623206 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4058119462 0 0) 0x7f75e8141070 con 0x7f75e80704e0 2024-03-25T19:36:34.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.561+0000 7f75e67fc640 1 -- 172.21.15.44:0/4237623206 >> v1:172.21.15.44:6790/0 conn(0x7f75e8141570 legacy=0x7f75e81419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:34.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.561+0000 7f75e67fc640 1 -- 172.21.15.44:0/4237623206 >> v1:172.21.15.44:6789/0 conn(0x7f75e80ae6e0 legacy=0x7f75e81db790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:34.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.561+0000 7f75e67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:36:34.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.561+0000 7f75e67fc640 1 monclient: found mon.b 2024-03-25T19:36:34.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.561+0000 7f75e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:34.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.561+0000 7f75e67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:34.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.561+0000 7f75e67fc640 1 -- 172.21.15.44:0/4237623206 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e8142260 con 0x7f75e80704e0 2024-03-25T19:36:34.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.561+0000 7f75e67fc640 10 monclient: _finish_auth 0 2024-03-25T19:36:34.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.561+0000 7f75e67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:36:34.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.561+0000 7f75e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:34.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.561+0000 7f75efdd0640 5 monclient: authenticate success, global_id 4273 2024-03-25T19:36:34.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.561+0000 7f75efdd0640 10 monclient: _renew_subs 2024-03-25T19:36:34.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.561+0000 7f75efdd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:34.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.561+0000 7f75efdd0640 1 -- 172.21.15.44:0/4237623206 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e81e3360 con 0x7f75e80704e0 2024-03-25T19:36:34.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.561+0000 7f75efdd0640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:34.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.561+0000 7f75efdd0640 10 monclient: _renew_subs 2024-03-25T19:36:34.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.561+0000 7f75efdd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:34.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.561+0000 7f75efdd0640 1 -- 172.21.15.44:0/4237623206 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e81e3790 con 0x7f75e80704e0 2024-03-25T19:36:34.563 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.561+0000 7f75efdd0640 1 librados: init done 2024-03-25T19:36:34.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.562+0000 7f75efdd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:34.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.562+0000 7f75efdd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:34.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.562+0000 7f75efdd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:34.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.562+0000 7f75efdd0640 1 -- 172.21.15.44:0/4237623206 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75e80b1c10 con 0x7f75e80704e0 2024-03-25T19:36:34.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.778+0000 7f75e67fc640 1 -- 172.21.15.44:0/4237623206 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75e00038a0 con 0x7f75e80704e0 2024-03-25T19:36:34.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.778+0000 7f75e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:34.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.778+0000 7f75e67fc640 1 -- 172.21.15.44:0/4237623206 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f75e0003cb0 con 0x7f75e80704e0 2024-03-25T19:36:34.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.778+0000 7f75e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:34.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.778+0000 7f75e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:34.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.778+0000 7f75e67fc640 10 monclient: dump: 2024-03-25T19:36:34.779 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:34.779 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:34.779 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:34.779 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:34.780 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:34.780 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:34.780 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:34.780 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:34.780 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:34.780 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:34.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.778+0000 7f75e7fff640 4 set_mon_vals no callback set 2024-03-25T19:36:34.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.838+0000 7f75e67fc640 1 -- 172.21.15.44:0/4237623206 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f75e0003e10 con 0x7f75e80704e0 2024-03-25T19:36:34.838 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.838+0000 7f75ed36b640 10 monclient: get_auth_request con 0x7f75d406dd60 auth_method 0 2024-03-25T19:36:34.838 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.839+0000 7f75e67fc640 10 client.4273.objecter ms_handle_connect 0x7f75d406dd60 2024-03-25T19:36:34.841 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.841+0000 7f75e67fc640 1 -- 172.21.15.44:0/4237623206 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f75e0054950 con 0x7f75e80704e0 2024-03-25T19:36:34.841 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.841+0000 7f75e67fc640 10 client.4273.objecter ms_dispatch 0x7f75e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:34.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.841+0000 7f75e67fc640 3 client.4273.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:34.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.841+0000 7f75e67fc640 3 client.4273.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:34.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.841+0000 7f75e67fc640 20 client.4273.objecter dump_active .. 0 homeless 2024-03-25T19:36:34.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.847+0000 7f75e67fc640 1 -- 172.21.15.44:0/4237623206 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f75e0050460 con 0x7f75e80704e0 2024-03-25T19:36:34.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.847+0000 7f75e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:34.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:34.847+0000 7f75e67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:35.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:35.102+0000 7f75efdd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:35.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:35.102+0000 7f75efdd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:35.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:35.102+0000 7f75efdd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:35.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:35.102+0000 7f75efdd0640 1 -- 172.21.15.44:0/4237623206 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75e8140ed0 con 0x7f75e80704e0 2024-03-25T19:36:35.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:35.130+0000 7f75e67fc640 1 -- 172.21.15.44:0/4237623206 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f75e8140ed0 con 0x7f75e80704e0 2024-03-25T19:36:35.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:35.130+0000 7f75e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:35.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:35.130+0000 7f75e67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:35.131 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:36:35.131 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:35.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:35.133+0000 7f75efdd0640 10 librados: watch_flush enter 2024-03-25T19:36:35.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:35.133+0000 7f75efdd0640 10 librados: watch_flush exit 2024-03-25T19:36:35.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:35.133+0000 7f75efdd0640 20 client.4273.objecter shutdown clearing up homeless session... 2024-03-25T19:36:35.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:35.133+0000 7f75efdd0640 10 client.4273.objecter successfully canceled tick 2024-03-25T19:36:35.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:35.133+0000 7f75efdd0640 1 -- 172.21.15.44:0/4237623206 >> v1:172.21.15.111:6800/3883215793 conn(0x7f75d406dd60 legacy=0x7f75d4070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:35.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:35.133+0000 7f75efdd0640 10 monclient: shutdown 2024-03-25T19:36:35.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:35.133+0000 7f75efdd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:35.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:35.133+0000 7f75efdd0640 1 -- 172.21.15.44:0/4237623206 >> v1:172.21.15.111:6789/0 conn(0x7f75e80704e0 legacy=0x7f75e81db080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:35.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:35.133+0000 7f75efdd0640 1 -- 172.21.15.44:0/4237623206 shutdown_connections 2024-03-25T19:36:35.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:35.133+0000 7f75efdd0640 1 -- 172.21.15.44:0/4237623206 >> 172.21.15.44:0/4237623206 conn(0x7f75e806b630 msgr2=0x7f75e80b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:35.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:35.133+0000 7f75efdd0640 1 -- 172.21.15.44:0/4237623206 shutdown_connections 2024-03-25T19:36:35.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:35.133+0000 7f75efdd0640 1 -- 172.21.15.44:0/4237623206 wait complete. 2024-03-25T19:36:35.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:35.133+0000 7f75efdd0640 1 librados: shutdown 2024-03-25T19:36:35.145 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:41.951 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:42.068 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.069+0000 7fa75d53e640 10 monclient: get_monmap_and_config 2024-03-25T19:36:42.068 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.069+0000 7fa75d53e640 10 monclient: build_initial_monmap 2024-03-25T19:36:42.068 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.069+0000 7fa75d53e640 10 monclient: monmap: 2024-03-25T19:36:42.068 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:36:42.068 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:42.068 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:36:42.069233+0000 2024-03-25T19:36:42.068 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:36:42.069233+0000 2024-03-25T19:36:42.068 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:42.068 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:42.069 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:36:42.069 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:36:42.069 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:36:42.069 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:42.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.069+0000 7fa75d53e640 1 Processor -- start 2024-03-25T19:36:42.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.069+0000 7fa75d53e640 1 -- start start 2024-03-25T19:36:42.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.069+0000 7fa75d53e640 10 monclient: init 2024-03-25T19:36:42.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.070+0000 7fa75d53e640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:42.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.070+0000 7fa75d53e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:42.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.070+0000 7fa75d53e640 10 monclient(hunting): picked mon.noname-a con 0x7fa7580a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:36:42.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.070+0000 7fa75d53e640 10 monclient(hunting): picked mon.noname-b con 0x7fa75813f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:36:42.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.070+0000 7fa75d53e640 10 monclient(hunting): picked mon.noname-c con 0x7fa7581430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:36:42.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.070+0000 7fa75d53e640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa758142520 con 0x7fa7580a98f0 2024-03-25T19:36:42.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.070+0000 7fa75d53e640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7580b28a0 con 0x7fa7581430c0 2024-03-25T19:36:42.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.070+0000 7fa75d53e640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7580b2a30 con 0x7fa75813f710 2024-03-25T19:36:42.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.070+0000 7fa75d53e640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:42.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.070+0000 7fa75d53e640 10 monclient(hunting): authenticate will time out at 5466.240443s 2024-03-25T19:36:42.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.073+0000 7fa7577fe640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fa7581430c0 0x7fa75814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:39900/0 (socket says 172.21.15.44:39900) 2024-03-25T19:36:42.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.073+0000 7fa7577fe640 1 -- 172.21.15.44:0/3421810722 learned_addr learned my addr 172.21.15.44:0/3421810722 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:36:42.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.084+0000 7fa755ffb640 1 -- 172.21.15.44:0/3421810722 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3672420935 0 0) 0x7fa7580b28a0 con 0x7fa7581430c0 2024-03-25T19:36:42.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.084+0000 7fa755ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:42.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.084+0000 7fa755ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:42.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.084+0000 7fa755ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:42.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.084+0000 7fa755ffb640 10 monclient(hunting): my global_id is 4276 2024-03-25T19:36:42.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.084+0000 7fa755ffb640 1 -- 172.21.15.44:0/3421810722 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa740003520 con 0x7fa7581430c0 2024-03-25T19:36:42.090 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.091+0000 7fa755ffb640 1 -- 172.21.15.44:0/3421810722 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478402480 0 0) 0x7fa7580b2a30 con 0x7fa75813f710 2024-03-25T19:36:42.090 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.091+0000 7fa755ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:42.090 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.091+0000 7fa755ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:42.090 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.091+0000 7fa755ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:42.090 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.091+0000 7fa755ffb640 10 monclient(hunting): my global_id is 4259 2024-03-25T19:36:42.090 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.091+0000 7fa755ffb640 1 -- 172.21.15.44:0/3421810722 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7580b28a0 con 0x7fa75813f710 2024-03-25T19:36:42.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.101+0000 7fa755ffb640 1 -- 172.21.15.44:0/3421810722 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1673101079 0 0) 0x7fa7580b28a0 con 0x7fa75813f710 2024-03-25T19:36:42.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.101+0000 7fa755ffb640 1 -- 172.21.15.44:0/3421810722 >> v1:172.21.15.111:6789/0 conn(0x7fa7581430c0 legacy=0x7fa75814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:42.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.101+0000 7fa755ffb640 1 -- 172.21.15.44:0/3421810722 >> v1:172.21.15.44:6789/0 conn(0x7fa7580a98f0 legacy=0x7fa7580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:42.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.101+0000 7fa755ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:36:42.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.101+0000 7fa755ffb640 1 monclient: found mon.noname-b 2024-03-25T19:36:42.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.101+0000 7fa755ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:42.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.102+0000 7fa755ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:36:42.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.102+0000 7fa755ffb640 1 -- 172.21.15.44:0/3421810722 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7580b44c0 con 0x7fa75813f710 2024-03-25T19:36:42.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.102+0000 7fa755ffb640 10 monclient: _finish_auth 0 2024-03-25T19:36:42.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.102+0000 7fa755ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:36:42.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.102+0000 7fa755ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:42.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.102+0000 7fa755ffb640 1 -- 172.21.15.44:0/3421810722 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fa74c001670 con 0x7fa75813f710 2024-03-25T19:36:42.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.102+0000 7fa75d53e640 5 monclient: authenticate success, global_id 4259 2024-03-25T19:36:42.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.102+0000 7fa75d53e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:42.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.102+0000 7fa755ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:42.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.102+0000 7fa755ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:42.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.102+0000 7fa755ffb640 10 monclient: dump: 2024-03-25T19:36:42.102 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:42.102 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:42.102 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:42.102 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:42.102 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:42.102 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:42.102 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:42.102 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:42.103 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:42.103 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:42.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.102+0000 7fa75d53e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:42.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.105+0000 7fa755ffb640 1 -- 172.21.15.44:0/3421810722 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa74c0017d0 con 0x7fa75813f710 2024-03-25T19:36:42.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.105+0000 7fa755ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:42.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.105+0000 7fa755ffb640 1 -- 172.21.15.44:0/3421810722 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fa74c001ba0 con 0x7fa75813f710 2024-03-25T19:36:42.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.105+0000 7fa75d53e640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:42.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.105+0000 7fa75d53e640 4 set_mon_vals no callback set 2024-03-25T19:36:42.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.105+0000 7fa755ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:42.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.105+0000 7fa755ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:42.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.105+0000 7fa755ffb640 10 monclient: dump: 2024-03-25T19:36:42.106 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:42.106 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:42.106 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:42.106 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:42.106 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:42.106 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:42.106 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:42.106 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:42.106 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:42.106 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:42.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.106+0000 7fa75d53e640 10 monclient: shutdown 2024-03-25T19:36:42.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.106+0000 7fa75d53e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:42.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.106+0000 7fa75d53e640 1 -- 172.21.15.44:0/3421810722 >> v1:172.21.15.44:6790/0 conn(0x7fa75813f710 legacy=0x7fa7581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:42.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.106+0000 7fa75d53e640 1 -- 172.21.15.44:0/3421810722 shutdown_connections 2024-03-25T19:36:42.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.106+0000 7fa75d53e640 1 -- 172.21.15.44:0/3421810722 >> 172.21.15.44:0/3421810722 conn(0x7fa7580b7ea0 msgr2=0x7fa7580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:42.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.106+0000 7fa75d53e640 1 -- 172.21.15.44:0/3421810722 shutdown_connections 2024-03-25T19:36:42.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.106+0000 7fa75d53e640 1 -- 172.21.15.44:0/3421810722 wait complete. 2024-03-25T19:36:42.108 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.107+0000 7fa75d53e640 10 monclient: build_initial_monmap 2024-03-25T19:36:42.108 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.107+0000 7fa75d53e640 10 monclient: monmap: 2024-03-25T19:36:42.108 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:36:42.108 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:42.108 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:36:42.108 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:36:42.109 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:42.109 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:36:42.109 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:36:42.109 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:36:42.109 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:36:42.109 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:42.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.107+0000 7fa75d53e640 1 librados: starting msgr at 2024-03-25T19:36:42.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.107+0000 7fa75d53e640 1 librados: starting objecter 2024-03-25T19:36:42.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.107+0000 7fa75d53e640 1 Processor -- start 2024-03-25T19:36:42.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.107+0000 7fa75d53e640 1 -- start start 2024-03-25T19:36:42.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.107+0000 7fa75d53e640 1 librados: setting wanted keys 2024-03-25T19:36:42.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.107+0000 7fa75d53e640 1 librados: calling monclient init 2024-03-25T19:36:42.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.107+0000 7fa75d53e640 10 monclient: init 2024-03-25T19:36:42.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.108+0000 7fa75d53e640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:42.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.108+0000 7fa75d53e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:42.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.108+0000 7fa75d53e640 10 monclient(hunting): picked mon.noname-a con 0x7fa7580a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:36:42.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.108+0000 7fa75d53e640 10 monclient(hunting): picked mon.noname-c con 0x7fa75813f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:36:42.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.108+0000 7fa75d53e640 10 monclient(hunting): picked mon.noname-b con 0x7fa7581430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:36:42.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.108+0000 7fa75d53e640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7581d78e0 con 0x7fa7580a98f0 2024-03-25T19:36:42.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.108+0000 7fa75d53e640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7581d7500 con 0x7fa7581430c0 2024-03-25T19:36:42.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.108+0000 7fa75d53e640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7581d7690 con 0x7fa75813f710 2024-03-25T19:36:42.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.108+0000 7fa75d53e640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:42.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.108+0000 7fa75d53e640 10 monclient(hunting): authenticate will time out at 5466.278365s 2024-03-25T19:36:42.124 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.125+0000 7fa7577fe640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fa7581430c0 0x7fa758138560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:39924/0 (socket says 172.21.15.44:39924) 2024-03-25T19:36:42.124 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.125+0000 7fa7577fe640 1 -- 172.21.15.44:0/3288175536 learned_addr learned my addr 172.21.15.44:0/3288175536 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:36:42.125 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.126+0000 7fa73f7fe640 10 client.?.objecter ms_handle_connect 0x7fa7581430c0 2024-03-25T19:36:42.125 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.126+0000 7fa73f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:42.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.130+0000 7fa73f7fe640 10 client.?.objecter ms_handle_connect 0x7fa75813f710 2024-03-25T19:36:42.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.130+0000 7fa73f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:42.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.130+0000 7fa73f7fe640 1 -- 172.21.15.44:0/3288175536 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832607724 0 0) 0x7fa7581d7500 con 0x7fa7581430c0 2024-03-25T19:36:42.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.130+0000 7fa73f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:42.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.130+0000 7fa73f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:42.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.130+0000 7fa73f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:42.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.130+0000 7fa73f7fe640 10 monclient(hunting): my global_id is 4279 2024-03-25T19:36:42.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.130+0000 7fa73f7fe640 1 -- 172.21.15.44:0/3288175536 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa744004eb0 con 0x7fa7581430c0 2024-03-25T19:36:42.132 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.133+0000 7fa73f7fe640 10 client.?.objecter ms_handle_connect 0x7fa7580a98f0 2024-03-25T19:36:42.132 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.133+0000 7fa73f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:42.143 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.145+0000 7fa73f7fe640 1 -- 172.21.15.44:0/3288175536 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604039152 0 0) 0x7fa7581d7690 con 0x7fa75813f710 2024-03-25T19:36:42.143 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.145+0000 7fa73f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:42.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.145+0000 7fa73f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:42.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.145+0000 7fa73f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:42.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.145+0000 7fa73f7fe640 10 monclient(hunting): my global_id is 4262 2024-03-25T19:36:42.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.145+0000 7fa73f7fe640 1 -- 172.21.15.44:0/3288175536 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7581d7500 con 0x7fa75813f710 2024-03-25T19:36:42.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.150+0000 7fa73f7fe640 1 -- 172.21.15.44:0/3288175536 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2925147643 0 0) 0x7fa744004eb0 con 0x7fa7581430c0 2024-03-25T19:36:42.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.150+0000 7fa73f7fe640 1 -- 172.21.15.44:0/3288175536 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7581d7690 con 0x7fa7581430c0 2024-03-25T19:36:42.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.150+0000 7fa73f7fe640 1 -- 172.21.15.44:0/3288175536 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fa750001b30 con 0x7fa7581430c0 2024-03-25T19:36:42.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.150+0000 7fa73f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:42.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.150+0000 7fa73f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:42.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.150+0000 7fa73f7fe640 10 monclient(hunting): dump: 2024-03-25T19:36:42.150 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:42.150 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:42.150 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:42.150 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:42.150 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:42.150 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:42.150 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:42.150 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:42.150 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:42.150 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:42.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.150+0000 7fa73f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:42.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.152+0000 7fa73f7fe640 1 -- 172.21.15.44:0/3288175536 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1791464414 0 0) 0x7fa7581d7500 con 0x7fa75813f710 2024-03-25T19:36:42.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.152+0000 7fa73f7fe640 1 -- 172.21.15.44:0/3288175536 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa744004eb0 con 0x7fa75813f710 2024-03-25T19:36:42.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.152+0000 7fa73f7fe640 1 -- 172.21.15.44:0/3288175536 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fa74c001ae0 con 0x7fa75813f710 2024-03-25T19:36:42.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.152+0000 7fa73f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:42.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.152+0000 7fa73f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:42.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.152+0000 7fa73f7fe640 10 monclient(hunting): dump: 2024-03-25T19:36:42.151 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:42.151 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:42.151 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:42.152 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:42.152 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:42.152 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:42.152 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:42.152 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:42.152 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:42.152 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:42.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.153+0000 7fa73f7fe640 1 -- 172.21.15.44:0/3288175536 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2301848942 0 0) 0x7fa7581d78e0 con 0x7fa7580a98f0 2024-03-25T19:36:42.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.153+0000 7fa73f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:42.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.153+0000 7fa73f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:42.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.153+0000 7fa73f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:42.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.153+0000 7fa73f7fe640 10 monclient(hunting): my global_id is 4293 2024-03-25T19:36:42.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.153+0000 7fa73f7fe640 1 -- 172.21.15.44:0/3288175536 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7581d7500 con 0x7fa7580a98f0 2024-03-25T19:36:42.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.165+0000 7fa73f7fe640 1 -- 172.21.15.44:0/3288175536 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1322821901 0 0) 0x7fa744004eb0 con 0x7fa75813f710 2024-03-25T19:36:42.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.165+0000 7fa73f7fe640 1 -- 172.21.15.44:0/3288175536 >> v1:172.21.15.111:6789/0 conn(0x7fa7581430c0 legacy=0x7fa758138560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:42.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.165+0000 7fa73f7fe640 1 -- 172.21.15.44:0/3288175536 >> v1:172.21.15.44:6789/0 conn(0x7fa7580a98f0 legacy=0x7fa758134de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:42.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.165+0000 7fa73f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:36:42.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.165+0000 7fa73f7fe640 1 monclient: found mon.c 2024-03-25T19:36:42.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.165+0000 7fa73f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:42.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.165+0000 7fa73f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:36:42.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.165+0000 7fa73f7fe640 1 -- 172.21.15.44:0/3288175536 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7581377c0 con 0x7fa75813f710 2024-03-25T19:36:42.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.165+0000 7fa73f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:36:42.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.165+0000 7fa73f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:36:42.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.165+0000 7fa73f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:42.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.165+0000 7fa75d53e640 5 monclient: authenticate success, global_id 4262 2024-03-25T19:36:42.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.165+0000 7fa75d53e640 10 monclient: _renew_subs 2024-03-25T19:36:42.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.165+0000 7fa75d53e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:36:42.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.165+0000 7fa75d53e640 1 -- 172.21.15.44:0/3288175536 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa758137a70 con 0x7fa75813f710 2024-03-25T19:36:42.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.166+0000 7fa75d53e640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:42.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.166+0000 7fa75d53e640 10 monclient: _renew_subs 2024-03-25T19:36:42.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.166+0000 7fa75d53e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:36:42.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.166+0000 7fa75d53e640 1 -- 172.21.15.44:0/3288175536 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7581341f0 con 0x7fa75813f710 2024-03-25T19:36:42.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.166+0000 7fa75d53e640 1 librados: init done 2024-03-25T19:36:42.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.167+0000 7fa75d53e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:42.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.167+0000 7fa75d53e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:42.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.167+0000 7fa75d53e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:36:42.167 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.167+0000 7fa75d53e640 1 -- 172.21.15.44:0/3288175536 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7580abbe0 con 0x7fa75813f710 2024-03-25T19:36:42.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.380+0000 7fa73f7fe640 1 -- 172.21.15.44:0/3288175536 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa74c0038a0 con 0x7fa75813f710 2024-03-25T19:36:42.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.380+0000 7fa73f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:42.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.380+0000 7fa7557fa640 4 set_mon_vals no callback set 2024-03-25T19:36:42.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.482+0000 7fa73f7fe640 1 -- 172.21.15.44:0/3288175536 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fa74c003bb0 con 0x7fa75813f710 2024-03-25T19:36:42.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.482+0000 7fa73f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:42.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.482+0000 7fa73f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:42.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.482+0000 7fa73f7fe640 10 monclient: dump: 2024-03-25T19:36:42.481 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:42.481 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:42.481 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:42.481 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:42.481 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:42.482 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:42.482 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:42.482 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:42.482 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:42.482 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:42.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.483+0000 7fa73f7fe640 1 -- 172.21.15.44:0/3288175536 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7fa74c019b80 con 0x7fa75813f710 2024-03-25T19:36:42.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.483+0000 7fa756ffd640 10 monclient: get_auth_request con 0x7fa74406df30 auth_method 0 2024-03-25T19:36:42.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.483+0000 7fa73f7fe640 10 client.4262.objecter ms_handle_connect 0x7fa74406df30 2024-03-25T19:36:42.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.488+0000 7fa73f7fe640 1 -- 172.21.15.44:0/3288175536 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7fa74c0536e0 con 0x7fa75813f710 2024-03-25T19:36:42.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.488+0000 7fa73f7fe640 10 client.4262.objecter ms_dispatch 0x7fa758000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:42.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.489+0000 7fa73f7fe640 3 client.4262.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:42.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.489+0000 7fa73f7fe640 3 client.4262.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:42.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.489+0000 7fa73f7fe640 20 client.4262.objecter dump_active .. 0 homeless 2024-03-25T19:36:42.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.489+0000 7fa73f7fe640 1 -- 172.21.15.44:0/3288175536 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa74c04f1f0 con 0x7fa75813f710 2024-03-25T19:36:42.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.489+0000 7fa73f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:42.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.489+0000 7fa73f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:42.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.728+0000 7fa75d53e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:42.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.728+0000 7fa75d53e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:42.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.728+0000 7fa75d53e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:36:42.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.728+0000 7fa75d53e640 1 -- 172.21.15.44:0/3288175536 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7580627b0 con 0x7fa75813f710 2024-03-25T19:36:42.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.754+0000 7fa73f7fe640 1 -- 172.21.15.44:0/3288175536 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7fa74c051120 con 0x7fa75813f710 2024-03-25T19:36:42.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.754+0000 7fa73f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:42.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.754+0000 7fa73f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:42.753 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:36:42.754 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:42.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.757+0000 7fa75d53e640 10 librados: watch_flush enter 2024-03-25T19:36:42.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.757+0000 7fa75d53e640 10 librados: watch_flush exit 2024-03-25T19:36:42.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.757+0000 7fa75d53e640 20 client.4262.objecter shutdown clearing up homeless session... 2024-03-25T19:36:42.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.757+0000 7fa75d53e640 10 client.4262.objecter successfully canceled tick 2024-03-25T19:36:42.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.757+0000 7fa75d53e640 1 -- 172.21.15.44:0/3288175536 >> v1:172.21.15.111:6800/3883215793 conn(0x7fa74406df30 legacy=0x7fa7440703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:42.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.757+0000 7fa75d53e640 10 monclient: shutdown 2024-03-25T19:36:42.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.757+0000 7fa75d53e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:42.757 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.757+0000 7fa75d53e640 1 -- 172.21.15.44:0/3288175536 >> v1:172.21.15.44:6790/0 conn(0x7fa75813f710 legacy=0x7fa7581d7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:42.757 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.757+0000 7fa75d53e640 1 -- 172.21.15.44:0/3288175536 shutdown_connections 2024-03-25T19:36:42.757 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.757+0000 7fa75d53e640 1 -- 172.21.15.44:0/3288175536 >> 172.21.15.44:0/3288175536 conn(0x7fa7580b7ea0 msgr2=0x7fa758074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:42.757 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.757+0000 7fa75d53e640 1 -- 172.21.15.44:0/3288175536 shutdown_connections 2024-03-25T19:36:42.757 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.757+0000 7fa75d53e640 1 -- 172.21.15.44:0/3288175536 wait complete. 2024-03-25T19:36:42.757 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:42.757+0000 7fa75d53e640 1 librados: shutdown 2024-03-25T19:36:42.767 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:43.817 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:01:34.719 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:43.820 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:01:34.722 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:43.865 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:01:34.745 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:43.870 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:01:34.772 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:43.875 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:01:34.777 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:43.877 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:01:34.757 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:43.886 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:01:34.788 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:43.922 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:01:34.801 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:43.939 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:01:34.841 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:49.574 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:49.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.688+0000 7f3beea15640 10 monclient: get_monmap_and_config 2024-03-25T19:36:49.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.688+0000 7f3beea15640 10 monclient: build_initial_monmap 2024-03-25T19:36:49.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.688+0000 7f3beea15640 10 monclient: monmap: 2024-03-25T19:36:49.688 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:36:49.688 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:49.688 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:36:49.689320+0000 2024-03-25T19:36:49.688 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:36:49.689320+0000 2024-03-25T19:36:49.688 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:49.688 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:49.689 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:36:49.689 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:36:49.689 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:36:49.689 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:49.689 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.689+0000 7f3beea15640 1 Processor -- start 2024-03-25T19:36:49.689 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.689+0000 7f3beea15640 1 -- start start 2024-03-25T19:36:49.689 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.689+0000 7f3beea15640 10 monclient: init 2024-03-25T19:36:49.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.690+0000 7f3beea15640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:49.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.690+0000 7f3beea15640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:36:49.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.690+0000 7f3beea15640 10 monclient(hunting): picked mon.noname-c con 0x7f3be80a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:36:49.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.690+0000 7f3beea15640 10 monclient(hunting): picked mon.noname-a con 0x7f3be813f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:36:49.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.690+0000 7f3beea15640 10 monclient(hunting): picked mon.noname-b con 0x7f3be81430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:36:49.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.690+0000 7f3beea15640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be8142520 con 0x7f3be813f710 2024-03-25T19:36:49.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.690+0000 7f3beea15640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be80b28a0 con 0x7f3be80a98f0 2024-03-25T19:36:49.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.690+0000 7f3beea15640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be80b2a30 con 0x7f3be81430c0 2024-03-25T19:36:49.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.690+0000 7f3beea15640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:49.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.690+0000 7f3beea15640 10 monclient(hunting): authenticate will time out at 5473.861792s 2024-03-25T19:36:49.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.703+0000 7f3be77fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f3be813f710 0x7f3be81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:38484/0 (socket says 172.21.15.44:38484) 2024-03-25T19:36:49.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.703+0000 7f3be77fe640 1 -- 172.21.15.44:0/3062642750 learned_addr learned my addr 172.21.15.44:0/3062642750 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:36:49.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.713+0000 7f3be6ffd640 1 -- 172.21.15.44:0/3062642750 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2848087210 0 0) 0x7f3be8142520 con 0x7f3be813f710 2024-03-25T19:36:49.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.713+0000 7f3be6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:49.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.713+0000 7f3be6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:49.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.713+0000 7f3be6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:49.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.713+0000 7f3be6ffd640 10 monclient(hunting): my global_id is 4296 2024-03-25T19:36:49.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.713+0000 7f3be6ffd640 1 -- 172.21.15.44:0/3062642750 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bd0003520 con 0x7f3be813f710 2024-03-25T19:36:49.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.720+0000 7f3be6ffd640 1 -- 172.21.15.44:0/3062642750 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942127562 0 0) 0x7f3be80b28a0 con 0x7f3be80a98f0 2024-03-25T19:36:49.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.720+0000 7f3be6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:49.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.720+0000 7f3be6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:49.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.720+0000 7f3be6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:49.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.720+0000 7f3be6ffd640 10 monclient(hunting): my global_id is 4282 2024-03-25T19:36:49.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.720+0000 7f3be6ffd640 1 -- 172.21.15.44:0/3062642750 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be8142520 con 0x7f3be80a98f0 2024-03-25T19:36:49.725 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.725+0000 7f3be6ffd640 1 -- 172.21.15.44:0/3062642750 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700026667 0 0) 0x7f3be80b2a30 con 0x7f3be81430c0 2024-03-25T19:36:49.725 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.725+0000 7f3be6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:49.725 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.725+0000 7f3be6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:49.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.725+0000 7f3be6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:49.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.725+0000 7f3be6ffd640 10 monclient(hunting): my global_id is 4265 2024-03-25T19:36:49.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.725+0000 7f3be6ffd640 1 -- 172.21.15.44:0/3062642750 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be80b28a0 con 0x7f3be81430c0 2024-03-25T19:36:49.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.729+0000 7f3be6ffd640 1 -- 172.21.15.44:0/3062642750 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3523403888 0 0) 0x7f3be8142520 con 0x7f3be80a98f0 2024-03-25T19:36:49.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.729+0000 7f3be6ffd640 1 -- 172.21.15.44:0/3062642750 >> v1:172.21.15.44:6790/0 conn(0x7f3be81430c0 legacy=0x7f3be814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:49.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.729+0000 7f3be6ffd640 1 -- 172.21.15.44:0/3062642750 >> v1:172.21.15.44:6789/0 conn(0x7f3be813f710 legacy=0x7f3be81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:49.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.729+0000 7f3be6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:36:49.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.729+0000 7f3be6ffd640 1 monclient: found mon.noname-c 2024-03-25T19:36:49.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.729+0000 7f3be6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:49.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.729+0000 7f3be6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:36:49.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.729+0000 7f3be6ffd640 1 -- 172.21.15.44:0/3062642750 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be80b44c0 con 0x7f3be80a98f0 2024-03-25T19:36:49.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.729+0000 7f3be6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:36:49.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.729+0000 7f3be6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:36:49.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.729+0000 7f3be6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:49.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.729+0000 7f3be6ffd640 1 -- 172.21.15.44:0/3062642750 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3be00019f0 con 0x7f3be80a98f0 2024-03-25T19:36:49.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.729+0000 7f3be6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:49.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.729+0000 7f3be6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:49.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.729+0000 7f3be6ffd640 10 monclient: dump: 2024-03-25T19:36:49.731 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:49.731 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:49.731 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:49.731 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:49.731 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:49.731 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:49.731 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:49.731 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:49.731 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:49.731 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:49.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.729+0000 7f3beea15640 5 monclient: authenticate success, global_id 4282 2024-03-25T19:36:49.732 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.729+0000 7f3beea15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:49.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.753+0000 7f3be6ffd640 1 -- 172.21.15.44:0/3062642750 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3be0001b50 con 0x7f3be80a98f0 2024-03-25T19:36:49.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.753+0000 7f3be6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:49.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.753+0000 7f3be6ffd640 1 -- 172.21.15.44:0/3062642750 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3be0001f60 con 0x7f3be80a98f0 2024-03-25T19:36:49.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.753+0000 7f3beea15640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:49.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.753+0000 7f3beea15640 4 set_mon_vals no callback set 2024-03-25T19:36:49.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.753+0000 7f3be6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:49.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.753+0000 7f3be6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:49.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.753+0000 7f3be6ffd640 10 monclient: dump: 2024-03-25T19:36:49.753 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:49.753 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:49.754 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:49.754 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:49.754 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:49.754 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:49.754 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:49.754 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:49.754 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:49.754 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:49.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.753+0000 7f3beea15640 10 monclient: shutdown 2024-03-25T19:36:49.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.753+0000 7f3beea15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:49.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.753+0000 7f3beea15640 1 -- 172.21.15.44:0/3062642750 >> v1:172.21.15.111:6789/0 conn(0x7f3be80a98f0 legacy=0x7f3be80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:49.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.754+0000 7f3beea15640 1 -- 172.21.15.44:0/3062642750 shutdown_connections 2024-03-25T19:36:49.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.754+0000 7f3beea15640 1 -- 172.21.15.44:0/3062642750 >> 172.21.15.44:0/3062642750 conn(0x7f3be80b7ea0 msgr2=0x7f3be80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:49.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.754+0000 7f3beea15640 1 -- 172.21.15.44:0/3062642750 shutdown_connections 2024-03-25T19:36:49.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.754+0000 7f3beea15640 1 -- 172.21.15.44:0/3062642750 wait complete. 2024-03-25T19:36:49.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.754+0000 7f3beea15640 10 monclient: build_initial_monmap 2024-03-25T19:36:49.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.754+0000 7f3beea15640 10 monclient: monmap: 2024-03-25T19:36:49.756 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:36:49.756 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:49.757 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:36:49.757 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:36:49.757 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:49.757 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:36:49.757 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:36:49.757 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:36:49.757 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:36:49.757 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:49.757 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.755+0000 7f3beea15640 1 librados: starting msgr at 2024-03-25T19:36:49.757 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.755+0000 7f3beea15640 1 librados: starting objecter 2024-03-25T19:36:49.757 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.755+0000 7f3beea15640 1 Processor -- start 2024-03-25T19:36:49.757 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.755+0000 7f3beea15640 1 -- start start 2024-03-25T19:36:49.757 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.755+0000 7f3beea15640 1 librados: setting wanted keys 2024-03-25T19:36:49.757 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.755+0000 7f3beea15640 1 librados: calling monclient init 2024-03-25T19:36:49.758 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.755+0000 7f3beea15640 10 monclient: init 2024-03-25T19:36:49.758 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.755+0000 7f3beea15640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:49.758 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.755+0000 7f3beea15640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:49.758 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.755+0000 7f3beea15640 10 monclient(hunting): picked mon.noname-a con 0x7f3be80a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:36:49.758 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.756+0000 7f3beea15640 10 monclient(hunting): picked mon.noname-c con 0x7f3be813f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:36:49.758 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.756+0000 7f3beea15640 10 monclient(hunting): picked mon.noname-b con 0x7f3be81430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:36:49.758 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.756+0000 7f3beea15640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be8139330 con 0x7f3be80a98f0 2024-03-25T19:36:49.758 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.756+0000 7f3beea15640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be80abe70 con 0x7f3be81430c0 2024-03-25T19:36:49.758 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.756+0000 7f3beea15640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be80ac150 con 0x7f3be813f710 2024-03-25T19:36:49.758 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.756+0000 7f3beea15640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:49.758 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.756+0000 7f3beea15640 10 monclient(hunting): authenticate will time out at 5473.927022s 2024-03-25T19:36:49.763 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.764+0000 7f3be77fe640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f3be813f710 0x7f3be80ac570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:38126/0 (socket says 172.21.15.44:38126) 2024-03-25T19:36:49.763 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.764+0000 7f3be77fe640 1 -- 172.21.15.44:0/3849036829 learned_addr learned my addr 172.21.15.44:0/3849036829 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:36:49.766 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.766+0000 7f3be4ff9640 10 client.?.objecter ms_handle_connect 0x7f3be813f710 2024-03-25T19:36:49.766 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.766+0000 7f3be4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:49.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.770+0000 7f3be4ff9640 10 client.?.objecter ms_handle_connect 0x7f3be81430c0 2024-03-25T19:36:49.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.770+0000 7f3be4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:49.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.779+0000 7f3be4ff9640 1 -- 172.21.15.44:0/3849036829 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1459250915 0 0) 0x7f3be80ac150 con 0x7f3be813f710 2024-03-25T19:36:49.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.779+0000 7f3be4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:49.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.779+0000 7f3be4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:49.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.779+0000 7f3be4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:49.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.779+0000 7f3be4ff9640 10 monclient(hunting): my global_id is 4268 2024-03-25T19:36:49.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.779+0000 7f3be4ff9640 1 -- 172.21.15.44:0/3849036829 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bd4004eb0 con 0x7f3be813f710 2024-03-25T19:36:49.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.780+0000 7f3be4ff9640 1 -- 172.21.15.44:0/3849036829 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 260577851 0 0) 0x7f3be80abe70 con 0x7f3be81430c0 2024-03-25T19:36:49.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.780+0000 7f3be4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:49.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.780+0000 7f3be4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:49.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.780+0000 7f3be4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:49.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.780+0000 7f3be4ff9640 10 monclient(hunting): my global_id is 4285 2024-03-25T19:36:49.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.780+0000 7f3be4ff9640 1 -- 172.21.15.44:0/3849036829 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be80ac150 con 0x7f3be81430c0 2024-03-25T19:36:49.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.782+0000 7f3be4ff9640 10 client.?.objecter ms_handle_connect 0x7f3be80a98f0 2024-03-25T19:36:49.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.782+0000 7f3be4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:49.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.793+0000 7f3be4ff9640 1 -- 172.21.15.44:0/3849036829 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 857793002 0 0) 0x7f3be80ac150 con 0x7f3be81430c0 2024-03-25T19:36:49.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.793+0000 7f3be4ff9640 1 -- 172.21.15.44:0/3849036829 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3be80abe70 con 0x7f3be81430c0 2024-03-25T19:36:49.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.793+0000 7f3be4ff9640 1 -- 172.21.15.44:0/3849036829 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3bd8001850 con 0x7f3be81430c0 2024-03-25T19:36:49.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.793+0000 7f3be4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:49.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.793+0000 7f3be4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:49.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.793+0000 7f3be4ff9640 10 monclient(hunting): dump: 2024-03-25T19:36:49.793 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:49.793 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:49.794 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:49.794 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:49.794 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:49.794 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:49.794 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:49.794 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:49.794 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:49.794 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:49.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.793+0000 7f3be4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:49.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.799+0000 7f3be4ff9640 1 -- 172.21.15.44:0/3849036829 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4064974845 0 0) 0x7f3be80abe70 con 0x7f3be81430c0 2024-03-25T19:36:49.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.799+0000 7f3be4ff9640 1 -- 172.21.15.44:0/3849036829 >> v1:172.21.15.44:6790/0 conn(0x7f3be813f710 legacy=0x7f3be80ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:49.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.799+0000 7f3be4ff9640 1 -- 172.21.15.44:0/3849036829 >> v1:172.21.15.44:6789/0 conn(0x7f3be80a98f0 legacy=0x7f3be81389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:49.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.799+0000 7f3be4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:36:49.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.799+0000 7f3be4ff9640 1 monclient: found mon.b 2024-03-25T19:36:49.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.799+0000 7f3be4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:49.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.799+0000 7f3be4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:49.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.799+0000 7f3be4ff9640 1 -- 172.21.15.44:0/3849036829 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be80acc80 con 0x7f3be81430c0 2024-03-25T19:36:49.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.799+0000 7f3be4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:36:49.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.799+0000 7f3be4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:36:49.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.799+0000 7f3be4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:49.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.799+0000 7f3beea15640 5 monclient: authenticate success, global_id 4285 2024-03-25T19:36:49.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.799+0000 7f3beea15640 10 monclient: _renew_subs 2024-03-25T19:36:49.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.799+0000 7f3beea15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:49.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.799+0000 7f3beea15640 1 -- 172.21.15.44:0/3849036829 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3be8137e30 con 0x7f3be81430c0 2024-03-25T19:36:49.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.800+0000 7f3beea15640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:49.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.800+0000 7f3beea15640 10 monclient: _renew_subs 2024-03-25T19:36:49.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.800+0000 7f3beea15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:49.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.800+0000 7f3beea15640 1 -- 172.21.15.44:0/3849036829 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3be8136880 con 0x7f3be81430c0 2024-03-25T19:36:49.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.800+0000 7f3beea15640 1 librados: init done 2024-03-25T19:36:49.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.800+0000 7f3beea15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:49.802 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.800+0000 7f3beea15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:49.802 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.800+0000 7f3beea15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:49.802 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:49.800+0000 7f3beea15640 1 -- 172.21.15.44:0/3849036829 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3be80627b0 con 0x7f3be81430c0 2024-03-25T19:36:50.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.082+0000 7f3be4ff9640 1 -- 172.21.15.44:0/3849036829 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3bd8001ca0 con 0x7f3be81430c0 2024-03-25T19:36:50.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.082+0000 7f3be4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:50.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.082+0000 7f3be4ff9640 1 -- 172.21.15.44:0/3849036829 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3bd80037b0 con 0x7f3be81430c0 2024-03-25T19:36:50.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.082+0000 7f3be4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:50.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.082+0000 7f3be4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:50.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.082+0000 7f3be4ff9640 10 monclient: dump: 2024-03-25T19:36:50.082 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:50.082 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:50.082 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:50.083 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:50.083 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:50.083 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:50.083 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:50.083 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:50.083 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:50.083 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:50.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.082+0000 7f3be67fc640 4 set_mon_vals no callback set 2024-03-25T19:36:50.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.082+0000 7f3be4ff9640 1 -- 172.21.15.44:0/3849036829 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f3bd80039d0 con 0x7f3be81430c0 2024-03-25T19:36:50.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.083+0000 7f3be7fff640 10 monclient: get_auth_request con 0x7f3bd406ddd0 auth_method 0 2024-03-25T19:36:50.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.083+0000 7f3be4ff9640 10 client.4285.objecter ms_handle_connect 0x7f3bd406ddd0 2024-03-25T19:36:50.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.087+0000 7f3be4ff9640 1 -- 172.21.15.44:0/3849036829 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f3bd80534c0 con 0x7f3be81430c0 2024-03-25T19:36:50.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.087+0000 7f3be4ff9640 10 client.4285.objecter ms_dispatch 0x7f3be8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:50.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.087+0000 7f3be4ff9640 3 client.4285.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:50.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.087+0000 7f3be4ff9640 3 client.4285.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:50.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.087+0000 7f3be4ff9640 20 client.4285.objecter dump_active .. 0 homeless 2024-03-25T19:36:50.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.087+0000 7f3be4ff9640 1 -- 172.21.15.44:0/3849036829 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3bd804efd0 con 0x7f3be81430c0 2024-03-25T19:36:50.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.087+0000 7f3be4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:50.088 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.087+0000 7f3be4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:50.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.343+0000 7f3beea15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:50.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.343+0000 7f3beea15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:50.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.343+0000 7f3beea15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:50.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.343+0000 7f3beea15640 1 -- 172.21.15.44:0/3849036829 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3be81390e0 con 0x7f3be81430c0 2024-03-25T19:36:50.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.375+0000 7f3be4ff9640 1 -- 172.21.15.44:0/3849036829 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f3bd8050f00 con 0x7f3be81430c0 2024-03-25T19:36:50.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.375+0000 7f3be4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:50.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.375+0000 7f3be4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:50.375 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:36:50.375 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:50.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.377+0000 7f3beea15640 10 librados: watch_flush enter 2024-03-25T19:36:50.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.377+0000 7f3beea15640 10 librados: watch_flush exit 2024-03-25T19:36:50.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.377+0000 7f3beea15640 20 client.4285.objecter shutdown clearing up homeless session... 2024-03-25T19:36:50.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.377+0000 7f3beea15640 10 client.4285.objecter successfully canceled tick 2024-03-25T19:36:50.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.377+0000 7f3beea15640 1 -- 172.21.15.44:0/3849036829 >> v1:172.21.15.111:6800/3883215793 conn(0x7f3bd406ddd0 legacy=0x7f3bd4070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:50.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.377+0000 7f3beea15640 10 monclient: shutdown 2024-03-25T19:36:50.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.377+0000 7f3beea15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:50.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.377+0000 7f3beea15640 1 -- 172.21.15.44:0/3849036829 >> v1:172.21.15.111:6789/0 conn(0x7f3be81430c0 legacy=0x7f3be81e2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:50.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.377+0000 7f3beea15640 1 -- 172.21.15.44:0/3849036829 shutdown_connections 2024-03-25T19:36:50.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.378+0000 7f3beea15640 1 -- 172.21.15.44:0/3849036829 >> 172.21.15.44:0/3849036829 conn(0x7f3be80b7ea0 msgr2=0x7f3be8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:50.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.378+0000 7f3beea15640 1 -- 172.21.15.44:0/3849036829 shutdown_connections 2024-03-25T19:36:50.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.378+0000 7f3beea15640 1 -- 172.21.15.44:0/3849036829 wait complete. 2024-03-25T19:36:50.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:50.378+0000 7f3beea15640 1 librados: shutdown 2024-03-25T19:36:50.390 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:57.196 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:57.313 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.313+0000 7fcdc88a8640 10 monclient: get_monmap_and_config 2024-03-25T19:36:57.314 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.313+0000 7fcdc88a8640 10 monclient: build_initial_monmap 2024-03-25T19:36:57.314 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.313+0000 7fcdc88a8640 10 monclient: monmap: 2024-03-25T19:36:57.314 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:36:57.314 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:57.314 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:36:57.314863+0000 2024-03-25T19:36:57.314 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:36:57.314863+0000 2024-03-25T19:36:57.314 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:57.314 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:57.314 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:36:57.314 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:36:57.314 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:36:57.314 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:57.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.314+0000 7fcdc88a8640 1 Processor -- start 2024-03-25T19:36:57.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.314+0000 7fcdc88a8640 1 -- start start 2024-03-25T19:36:57.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.314+0000 7fcdc88a8640 10 monclient: init 2024-03-25T19:36:57.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.315+0000 7fcdc88a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:57.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.315+0000 7fcdc88a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:36:57.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.315+0000 7fcdc88a8640 10 monclient(hunting): picked mon.noname-c con 0x7fcdc40a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:36:57.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.315+0000 7fcdc88a8640 10 monclient(hunting): picked mon.noname-b con 0x7fcdc413f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:36:57.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.315+0000 7fcdc88a8640 10 monclient(hunting): picked mon.noname-a con 0x7fcdc41430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:36:57.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.315+0000 7fcdc88a8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdc4142520 con 0x7fcdc41430c0 2024-03-25T19:36:57.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.315+0000 7fcdc88a8640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdc40b28a0 con 0x7fcdc40a98f0 2024-03-25T19:36:57.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.315+0000 7fcdc88a8640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdc40b2a30 con 0x7fcdc413f710 2024-03-25T19:36:57.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.315+0000 7fcdc88a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:57.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.315+0000 7fcdc88a8640 10 monclient(hunting): authenticate will time out at 5481.486002s 2024-03-25T19:36:57.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.326+0000 7fcdb9d9b640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fcdc413f710 0x7fcdc41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:42730/0 (socket says 172.21.15.44:42730) 2024-03-25T19:36:57.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.326+0000 7fcdb9d9b640 1 -- 172.21.15.44:0/2338513574 learned_addr learned my addr 172.21.15.44:0/2338513574 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:36:57.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.334+0000 7fcdc1d9b640 1 -- 172.21.15.44:0/2338513574 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281327773 0 0) 0x7fcdc40b2a30 con 0x7fcdc413f710 2024-03-25T19:36:57.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.334+0000 7fcdc1d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:36:57.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.334+0000 7fcdc1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:57.335 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.334+0000 7fcdc1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:57.335 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.334+0000 7fcdc1d9b640 10 monclient(hunting): my global_id is 4271 2024-03-25T19:36:57.335 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.334+0000 7fcdc1d9b640 1 -- 172.21.15.44:0/2338513574 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdb0003520 con 0x7fcdc413f710 2024-03-25T19:36:57.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.342+0000 7fcdc1d9b640 1 -- 172.21.15.44:0/2338513574 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312436111 0 0) 0x7fcdc40b28a0 con 0x7fcdc40a98f0 2024-03-25T19:36:57.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.342+0000 7fcdc1d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:36:57.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.342+0000 7fcdc1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:57.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.342+0000 7fcdc1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:57.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.342+0000 7fcdc1d9b640 10 monclient(hunting): my global_id is 4288 2024-03-25T19:36:57.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.342+0000 7fcdc1d9b640 1 -- 172.21.15.44:0/2338513574 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdc40b2a30 con 0x7fcdc40a98f0 2024-03-25T19:36:57.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.350+0000 7fcdc1d9b640 1 -- 172.21.15.44:0/2338513574 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1576624909 0 0) 0x7fcdb0003520 con 0x7fcdc413f710 2024-03-25T19:36:57.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.350+0000 7fcdc1d9b640 1 -- 172.21.15.44:0/2338513574 >> v1:172.21.15.111:6789/0 conn(0x7fcdc40a98f0 legacy=0x7fcdc40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:57.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.350+0000 7fcdc1d9b640 1 -- 172.21.15.44:0/2338513574 >> v1:172.21.15.44:6789/0 conn(0x7fcdc41430c0 legacy=0x7fcdc414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:57.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.351+0000 7fcdc1d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:36:57.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.351+0000 7fcdc1d9b640 1 monclient: found mon.noname-b 2024-03-25T19:36:57.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.351+0000 7fcdc1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:57.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.351+0000 7fcdc1d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:36:57.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.351+0000 7fcdc1d9b640 1 -- 172.21.15.44:0/2338513574 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc40b44c0 con 0x7fcdc413f710 2024-03-25T19:36:57.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.351+0000 7fcdc1d9b640 10 monclient: _finish_auth 0 2024-03-25T19:36:57.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.351+0000 7fcdc1d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:36:57.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.351+0000 7fcdc1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:57.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.351+0000 7fcdc1d9b640 1 -- 172.21.15.44:0/2338513574 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fcdac001690 con 0x7fcdc413f710 2024-03-25T19:36:57.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.351+0000 7fcdc1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:57.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.351+0000 7fcdc1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:57.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.351+0000 7fcdc1d9b640 10 monclient: dump: 2024-03-25T19:36:57.352 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:57.352 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:57.352 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:57.352 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:57.352 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:57.352 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:57.352 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:57.352 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:57.352 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:57.352 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:57.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.351+0000 7fcdc88a8640 5 monclient: authenticate success, global_id 4271 2024-03-25T19:36:57.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.351+0000 7fcdc88a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:57.364 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.364+0000 7fcdc1d9b640 1 -- 172.21.15.44:0/2338513574 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcdac0017f0 con 0x7fcdc413f710 2024-03-25T19:36:57.364 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.364+0000 7fcdc1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:57.364 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.364+0000 7fcdc1d9b640 1 -- 172.21.15.44:0/2338513574 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fcdac001c40 con 0x7fcdc413f710 2024-03-25T19:36:57.364 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.364+0000 7fcdc1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:57.364 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.364+0000 7fcdc1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:57.364 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.364+0000 7fcdc1d9b640 10 monclient: dump: 2024-03-25T19:36:57.364 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:57.364 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:57.364 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:57.364 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:57.364 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:57.365 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:57.365 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:57.365 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:57.365 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:57.365 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:57.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.364+0000 7fcdc88a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:57.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.364+0000 7fcdc88a8640 4 set_mon_vals no callback set 2024-03-25T19:36:57.366 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.365+0000 7fcdc88a8640 10 monclient: shutdown 2024-03-25T19:36:57.366 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.365+0000 7fcdc88a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:57.366 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.365+0000 7fcdc88a8640 1 -- 172.21.15.44:0/2338513574 >> v1:172.21.15.44:6790/0 conn(0x7fcdc413f710 legacy=0x7fcdc41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:57.366 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.365+0000 7fcdc88a8640 1 -- 172.21.15.44:0/2338513574 shutdown_connections 2024-03-25T19:36:57.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.365+0000 7fcdc88a8640 1 -- 172.21.15.44:0/2338513574 >> 172.21.15.44:0/2338513574 conn(0x7fcdc40b7ea0 msgr2=0x7fcdc40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:57.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.365+0000 7fcdc88a8640 1 -- 172.21.15.44:0/2338513574 shutdown_connections 2024-03-25T19:36:57.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.365+0000 7fcdc88a8640 1 -- 172.21.15.44:0/2338513574 wait complete. 2024-03-25T19:36:57.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.366+0000 7fcdc88a8640 10 monclient: build_initial_monmap 2024-03-25T19:36:57.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.366+0000 7fcdc88a8640 10 monclient: monmap: 2024-03-25T19:36:57.367 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:36:57.367 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:57.367 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:36:57.367 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:36:57.368 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:57.368 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:36:57.368 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:36:57.368 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:36:57.368 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:36:57.368 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:57.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.367+0000 7fcdc88a8640 1 librados: starting msgr at 2024-03-25T19:36:57.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.367+0000 7fcdc88a8640 1 librados: starting objecter 2024-03-25T19:36:57.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.367+0000 7fcdc88a8640 1 Processor -- start 2024-03-25T19:36:57.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.367+0000 7fcdc88a8640 1 -- start start 2024-03-25T19:36:57.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.367+0000 7fcdc88a8640 1 librados: setting wanted keys 2024-03-25T19:36:57.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.367+0000 7fcdc88a8640 1 librados: calling monclient init 2024-03-25T19:36:57.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.367+0000 7fcdc88a8640 10 monclient: init 2024-03-25T19:36:57.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.368+0000 7fcdc88a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:57.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.368+0000 7fcdc88a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:36:57.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.368+0000 7fcdc88a8640 10 monclient(hunting): picked mon.noname-b con 0x7fcdc40a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:36:57.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.368+0000 7fcdc88a8640 10 monclient(hunting): picked mon.noname-c con 0x7fcdc413f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:36:57.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.368+0000 7fcdc88a8640 10 monclient(hunting): picked mon.noname-a con 0x7fcdc41430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:36:57.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.368+0000 7fcdc88a8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdc406fbe0 con 0x7fcdc41430c0 2024-03-25T19:36:57.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.368+0000 7fcdc88a8640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdc414e8b0 con 0x7fcdc40a98f0 2024-03-25T19:36:57.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.368+0000 7fcdc88a8640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdc406fec0 con 0x7fcdc413f710 2024-03-25T19:36:57.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.368+0000 7fcdc88a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:57.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.368+0000 7fcdc88a8640 10 monclient(hunting): authenticate will time out at 5481.539332s 2024-03-25T19:36:57.377 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.377+0000 7fcdc259c640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fcdc40a98f0 0x7fcdc4070930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:57610/0 (socket says 172.21.15.44:57610) 2024-03-25T19:36:57.377 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.377+0000 7fcdc259c640 1 -- 172.21.15.44:0/2637260539 learned_addr learned my addr 172.21.15.44:0/2637260539 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:36:57.377 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.378+0000 7fcdbb7fe640 10 client.?.objecter ms_handle_connect 0x7fcdc40a98f0 2024-03-25T19:36:57.377 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.378+0000 7fcdbb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:57.384 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.385+0000 7fcdbb7fe640 10 client.?.objecter ms_handle_connect 0x7fcdc41430c0 2024-03-25T19:36:57.385 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.385+0000 7fcdbb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:57.386 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.386+0000 7fcdbb7fe640 10 client.?.objecter ms_handle_connect 0x7fcdc413f710 2024-03-25T19:36:57.386 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.386+0000 7fcdbb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:57.398 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.399+0000 7fcdbb7fe640 1 -- 172.21.15.44:0/2637260539 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454378777 0 0) 0x7fcdc414e8b0 con 0x7fcdc40a98f0 2024-03-25T19:36:57.399 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.399+0000 7fcdbb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:57.399 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.399+0000 7fcdbb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:57.399 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.399+0000 7fcdbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:57.399 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.399+0000 7fcdbb7fe640 10 monclient(hunting): my global_id is 4291 2024-03-25T19:36:57.399 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.399+0000 7fcdbb7fe640 1 -- 172.21.15.44:0/2637260539 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcda4004eb0 con 0x7fcdc40a98f0 2024-03-25T19:36:57.399 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.400+0000 7fcdbb7fe640 1 -- 172.21.15.44:0/2637260539 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604675441 0 0) 0x7fcdc406fec0 con 0x7fcdc413f710 2024-03-25T19:36:57.399 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.400+0000 7fcdbb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:57.399 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.400+0000 7fcdbb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:57.400 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.400+0000 7fcdbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:57.400 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.400+0000 7fcdbb7fe640 10 monclient(hunting): my global_id is 4274 2024-03-25T19:36:57.400 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.400+0000 7fcdbb7fe640 1 -- 172.21.15.44:0/2637260539 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdc414e8b0 con 0x7fcdc413f710 2024-03-25T19:36:57.403 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.403+0000 7fcdbb7fe640 1 -- 172.21.15.44:0/2637260539 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482111919 0 0) 0x7fcdc406fbe0 con 0x7fcdc41430c0 2024-03-25T19:36:57.403 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.403+0000 7fcdbb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:57.403 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.403+0000 7fcdbb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:57.403 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.403+0000 7fcdbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:57.403 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.403+0000 7fcdbb7fe640 10 monclient(hunting): my global_id is 4305 2024-03-25T19:36:57.403 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.403+0000 7fcdbb7fe640 1 -- 172.21.15.44:0/2637260539 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdc406fec0 con 0x7fcdc41430c0 2024-03-25T19:36:57.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.408+0000 7fcdbb7fe640 1 -- 172.21.15.44:0/2637260539 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3042066554 0 0) 0x7fcda4004eb0 con 0x7fcdc40a98f0 2024-03-25T19:36:57.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.408+0000 7fcdbb7fe640 1 -- 172.21.15.44:0/2637260539 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcdc406fbe0 con 0x7fcdc40a98f0 2024-03-25T19:36:57.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.408+0000 7fcdbb7fe640 1 -- 172.21.15.44:0/2637260539 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fcdb4001a10 con 0x7fcdc40a98f0 2024-03-25T19:36:57.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.408+0000 7fcdbb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:57.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.408+0000 7fcdbb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:57.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.408+0000 7fcdbb7fe640 10 monclient(hunting): dump: 2024-03-25T19:36:57.408 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:57.408 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:57.408 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:57.408 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:57.408 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:57.408 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:57.408 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:57.409 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:57.409 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:57.409 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:57.409 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.408+0000 7fcdbb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:57.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.415+0000 7fcdbb7fe640 1 -- 172.21.15.44:0/2637260539 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2523926041 0 0) 0x7fcdc414e8b0 con 0x7fcdc413f710 2024-03-25T19:36:57.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.415+0000 7fcdbb7fe640 1 -- 172.21.15.44:0/2637260539 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcda4004eb0 con 0x7fcdc413f710 2024-03-25T19:36:57.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.415+0000 7fcdbb7fe640 1 -- 172.21.15.44:0/2637260539 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fcdac001b00 con 0x7fcdc413f710 2024-03-25T19:36:57.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.415+0000 7fcdbb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:57.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.415+0000 7fcdbb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:57.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.415+0000 7fcdbb7fe640 10 monclient(hunting): dump: 2024-03-25T19:36:57.416 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:57.416 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:57.416 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:57.416 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:57.416 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:57.416 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:57.416 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:57.416 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:57.416 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:57.416 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:57.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.415+0000 7fcdbb7fe640 1 -- 172.21.15.44:0/2637260539 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2484924834 0 0) 0x7fcdc406fec0 con 0x7fcdc41430c0 2024-03-25T19:36:57.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.415+0000 7fcdbb7fe640 1 -- 172.21.15.44:0/2637260539 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcdc414e8b0 con 0x7fcdc41430c0 2024-03-25T19:36:57.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.415+0000 7fcdbb7fe640 1 -- 172.21.15.44:0/2637260539 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fcdbc001a90 con 0x7fcdc41430c0 2024-03-25T19:36:57.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.415+0000 7fcdbb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:57.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.415+0000 7fcdbb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:57.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.415+0000 7fcdbb7fe640 10 monclient(hunting): dump: 2024-03-25T19:36:57.417 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:57.417 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:57.417 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:57.417 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:57.417 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:57.417 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:57.417 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:57.417 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:57.417 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:57.417 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:57.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.429+0000 7fcdbb7fe640 1 -- 172.21.15.44:0/2637260539 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2739874766 0 0) 0x7fcdc406fbe0 con 0x7fcdc40a98f0 2024-03-25T19:36:57.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.429+0000 7fcdbb7fe640 1 -- 172.21.15.44:0/2637260539 >> v1:172.21.15.44:6790/0 conn(0x7fcdc413f710 legacy=0x7fcdc414ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:57.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.429+0000 7fcdbb7fe640 1 -- 172.21.15.44:0/2637260539 >> v1:172.21.15.44:6789/0 conn(0x7fcdc41430c0 legacy=0x7fcdc4138f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:57.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.429+0000 7fcdbb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:36:57.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.429+0000 7fcdbb7fe640 1 monclient: found mon.b 2024-03-25T19:36:57.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.429+0000 7fcdbb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:57.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.429+0000 7fcdbb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:57.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.429+0000 7fcdbb7fe640 1 -- 172.21.15.44:0/2637260539 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc406e6a0 con 0x7fcdc40a98f0 2024-03-25T19:36:57.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.429+0000 7fcdbb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:36:57.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.429+0000 7fcdbb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:36:57.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.429+0000 7fcdbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:57.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.429+0000 7fcdc88a8640 5 monclient: authenticate success, global_id 4291 2024-03-25T19:36:57.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.429+0000 7fcdc88a8640 10 monclient: _renew_subs 2024-03-25T19:36:57.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.429+0000 7fcdc88a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:57.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.429+0000 7fcdc88a8640 1 -- 172.21.15.44:0/2637260539 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdc406e180 con 0x7fcdc40a98f0 2024-03-25T19:36:57.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.429+0000 7fcdc88a8640 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:57.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.429+0000 7fcdc88a8640 10 monclient: _renew_subs 2024-03-25T19:36:57.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.429+0000 7fcdc88a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:57.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.429+0000 7fcdc88a8640 1 -- 172.21.15.44:0/2637260539 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdc406f0f0 con 0x7fcdc40a98f0 2024-03-25T19:36:57.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.429+0000 7fcdc88a8640 1 librados: init done 2024-03-25T19:36:57.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.430+0000 7fcdc88a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:57.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.430+0000 7fcdc88a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:57.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.430+0000 7fcdc88a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:57.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.430+0000 7fcdc88a8640 1 -- 172.21.15.44:0/2637260539 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdc414e710 con 0x7fcdc40a98f0 2024-03-25T19:36:57.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.703+0000 7fcdbb7fe640 1 -- 172.21.15.44:0/2637260539 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcdb4001c50 con 0x7fcdc40a98f0 2024-03-25T19:36:57.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.703+0000 7fcdbb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:57.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.703+0000 7fcdbb7fe640 1 -- 172.21.15.44:0/2637260539 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fcdb40038c0 con 0x7fcdc40a98f0 2024-03-25T19:36:57.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.703+0000 7fcdbb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:57.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.703+0000 7fcdbb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:57.704 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.703+0000 7fcdbb7fe640 10 monclient: dump: 2024-03-25T19:36:57.704 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:36:57.704 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:36:57.704 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:57.704 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:36:57.704 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:57.704 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:36:57.704 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:36:57.704 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:36:57.704 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:36:57.704 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:36:57.704 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.703+0000 7fcdc0d99640 4 set_mon_vals no callback set 2024-03-25T19:36:57.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.704+0000 7fcdbb7fe640 1 -- 172.21.15.44:0/2637260539 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7fcdb4003aa0 con 0x7fcdc40a98f0 2024-03-25T19:36:57.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.704+0000 7fcdb9d9b640 10 monclient: get_auth_request con 0x7fcda406e0f0 auth_method 0 2024-03-25T19:36:57.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.705+0000 7fcdbb7fe640 10 client.4291.objecter ms_handle_connect 0x7fcda406e0f0 2024-03-25T19:36:57.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.709+0000 7fcdbb7fe640 1 -- 172.21.15.44:0/2637260539 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7fcdb40547b0 con 0x7fcdc40a98f0 2024-03-25T19:36:57.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.709+0000 7fcdbb7fe640 10 client.4291.objecter ms_dispatch 0x7fcdc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:57.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.709+0000 7fcdbb7fe640 3 client.4291.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:57.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.709+0000 7fcdbb7fe640 3 client.4291.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:57.710 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.709+0000 7fcdbb7fe640 20 client.4291.objecter dump_active .. 0 homeless 2024-03-25T19:36:57.710 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.710+0000 7fcdbb7fe640 1 -- 172.21.15.44:0/2637260539 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcdb40502c0 con 0x7fcdc40a98f0 2024-03-25T19:36:57.710 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.710+0000 7fcdbb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:57.710 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.710+0000 7fcdbb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:57.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.945+0000 7fcdc88a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:57.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.945+0000 7fcdc88a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:57.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.945+0000 7fcdc88a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:36:57.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.945+0000 7fcdc88a8640 1 -- 172.21.15.44:0/2637260539 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcdc40627b0 con 0x7fcdc40a98f0 2024-03-25T19:36:57.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.964+0000 7fcdbb7fe640 1 -- 172.21.15.44:0/2637260539 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7fcdb40521f0 con 0x7fcdc40a98f0 2024-03-25T19:36:57.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.964+0000 7fcdbb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:57.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.964+0000 7fcdbb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:57.964 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:36:57.965 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:57.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.967+0000 7fcdc88a8640 10 librados: watch_flush enter 2024-03-25T19:36:57.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.967+0000 7fcdc88a8640 10 librados: watch_flush exit 2024-03-25T19:36:57.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.967+0000 7fcdc88a8640 20 client.4291.objecter shutdown clearing up homeless session... 2024-03-25T19:36:57.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.967+0000 7fcdc88a8640 10 client.4291.objecter successfully canceled tick 2024-03-25T19:36:57.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.967+0000 7fcdc88a8640 1 -- 172.21.15.44:0/2637260539 >> v1:172.21.15.111:6800/3883215793 conn(0x7fcda406e0f0 legacy=0x7fcda40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:57.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.967+0000 7fcdc88a8640 10 monclient: shutdown 2024-03-25T19:36:57.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.967+0000 7fcdc88a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:57.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.967+0000 7fcdc88a8640 1 -- 172.21.15.44:0/2637260539 >> v1:172.21.15.111:6789/0 conn(0x7fcdc40a98f0 legacy=0x7fcdc4070930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:57.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.967+0000 7fcdc88a8640 1 -- 172.21.15.44:0/2637260539 shutdown_connections 2024-03-25T19:36:57.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.967+0000 7fcdc88a8640 1 -- 172.21.15.44:0/2637260539 >> 172.21.15.44:0/2637260539 conn(0x7fcdc40b7ea0 msgr2=0x7fcdc4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:57.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.967+0000 7fcdc88a8640 1 -- 172.21.15.44:0/2637260539 shutdown_connections 2024-03-25T19:36:57.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.967+0000 7fcdc88a8640 1 -- 172.21.15.44:0/2637260539 wait complete. 2024-03-25T19:36:57.969 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:36:57.968+0000 7fcdc88a8640 1 librados: shutdown 2024-03-25T19:36:57.980 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:03.687 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:01:54.588 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:03.689 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:01:54.591 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:03.757 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:01:54.659 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:03.792 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:01:54.672 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:03.804 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:01:54.706 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:03.804 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:01:54.684 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:03.824 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:01:54.726 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:03.847 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:01:54.727 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:03.870 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:01:54.772 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:04.786 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:04.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.888+0000 7f0087f90640 10 monclient: get_monmap_and_config 2024-03-25T19:37:04.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.888+0000 7f0087f90640 10 monclient: build_initial_monmap 2024-03-25T19:37:04.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.888+0000 7f0087f90640 10 monclient: monmap: 2024-03-25T19:37:04.888 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:37:04.888 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:04.888 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:37:04.889482+0000 2024-03-25T19:37:04.888 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:37:04.889482+0000 2024-03-25T19:37:04.888 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:04.888 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:04.888 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:37:04.889 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:37:04.889 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:37:04.889 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:04.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.889+0000 7f0087f90640 1 Processor -- start 2024-03-25T19:37:04.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.889+0000 7f0087f90640 1 -- start start 2024-03-25T19:37:04.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.889+0000 7f0087f90640 10 monclient: init 2024-03-25T19:37:04.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.890+0000 7f0087f90640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:04.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.890+0000 7f0087f90640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:37:04.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.890+0000 7f0087f90640 10 monclient(hunting): picked mon.noname-c con 0x7f00800b4f40 addr v1:172.21.15.111:6789/0 2024-03-25T19:37:04.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.890+0000 7f0087f90640 10 monclient(hunting): picked mon.noname-a con 0x7f008013fd90 addr v1:172.21.15.44:6789/0 2024-03-25T19:37:04.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.890+0000 7f0087f90640 10 monclient(hunting): picked mon.noname-b con 0x7f00800b4540 addr v1:172.21.15.44:6790/0 2024-03-25T19:37:04.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.890+0000 7f0087f90640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00800aa0c0 con 0x7f008013fd90 2024-03-25T19:37:04.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.890+0000 7f0087f90640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00800ab170 con 0x7f00800b4f40 2024-03-25T19:37:04.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.890+0000 7f0087f90640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00800a9ba0 con 0x7f00800b4540 2024-03-25T19:37:04.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.890+0000 7f0087f90640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:04.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.890+0000 7f0087f90640 10 monclient(hunting): authenticate will time out at 5489.061694s 2024-03-25T19:37:04.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.899+0000 7f008652d640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f00800b4540 0x7f008014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:39458/0 (socket says 172.21.15.44:39458) 2024-03-25T19:37:04.899 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.899+0000 7f008652d640 1 -- 172.21.15.44:0/968849561 learned_addr learned my addr 172.21.15.44:0/968849561 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:37:04.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.967+0000 7f0084d2a640 1 -- 172.21.15.44:0/968849561 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2721012611 0 0) 0x7f00800ab170 con 0x7f00800b4f40 2024-03-25T19:37:04.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.967+0000 7f0084d2a640 10 monclient(hunting): _negotiate 2024-03-25T19:37:04.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.967+0000 7f0084d2a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:04.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.967+0000 7f0084d2a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:04.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.967+0000 7f0084d2a640 10 monclient(hunting): my global_id is 4294 2024-03-25T19:37:04.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.967+0000 7f0084d2a640 1 -- 172.21.15.44:0/968849561 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f006c003520 con 0x7f00800b4f40 2024-03-25T19:37:04.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.975+0000 7f0084d2a640 1 -- 172.21.15.44:0/968849561 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3289881107 0 0) 0x7f00800aa0c0 con 0x7f008013fd90 2024-03-25T19:37:04.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.975+0000 7f0084d2a640 10 monclient(hunting): _negotiate 2024-03-25T19:37:04.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.975+0000 7f0084d2a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:04.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.975+0000 7f0084d2a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:04.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.975+0000 7f0084d2a640 10 monclient(hunting): my global_id is 4308 2024-03-25T19:37:04.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.975+0000 7f0084d2a640 1 -- 172.21.15.44:0/968849561 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00800ab170 con 0x7f008013fd90 2024-03-25T19:37:04.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.976+0000 7f0084d2a640 1 -- 172.21.15.44:0/968849561 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639464979 0 0) 0x7f00800a9ba0 con 0x7f00800b4540 2024-03-25T19:37:04.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.976+0000 7f0084d2a640 10 monclient(hunting): _negotiate 2024-03-25T19:37:04.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.976+0000 7f0084d2a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:04.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.976+0000 7f0084d2a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:04.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.976+0000 7f0084d2a640 10 monclient(hunting): my global_id is 4277 2024-03-25T19:37:04.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.976+0000 7f0084d2a640 1 -- 172.21.15.44:0/968849561 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00800aa0c0 con 0x7f00800b4540 2024-03-25T19:37:04.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.981+0000 7f0084d2a640 1 -- 172.21.15.44:0/968849561 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 828629817 0 0) 0x7f006c003520 con 0x7f00800b4f40 2024-03-25T19:37:04.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.981+0000 7f0084d2a640 1 -- 172.21.15.44:0/968849561 >> v1:172.21.15.44:6790/0 conn(0x7f00800b4540 legacy=0x7f008014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:04.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.981+0000 7f0084d2a640 1 -- 172.21.15.44:0/968849561 >> v1:172.21.15.44:6789/0 conn(0x7f008013fd90 legacy=0x7f00801423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:04.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.981+0000 7f0084d2a640 10 monclient: _finish_hunting 0 2024-03-25T19:37:04.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.981+0000 7f0084d2a640 1 monclient: found mon.noname-c 2024-03-25T19:37:04.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.981+0000 7f0084d2a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:04.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.981+0000 7f0084d2a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:37:04.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.981+0000 7f0084d2a640 1 -- 172.21.15.44:0/968849561 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00800a9680 con 0x7f00800b4f40 2024-03-25T19:37:04.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.981+0000 7f0084d2a640 10 monclient: _finish_auth 0 2024-03-25T19:37:04.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.981+0000 7f0084d2a640 10 monclient: _check_auth_tickets 2024-03-25T19:37:04.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.981+0000 7f0084d2a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:04.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.981+0000 7f0084d2a640 1 -- 172.21.15.44:0/968849561 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f007c001690 con 0x7f00800b4f40 2024-03-25T19:37:04.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.981+0000 7f0087f90640 5 monclient: authenticate success, global_id 4294 2024-03-25T19:37:04.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.981+0000 7f0087f90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:04.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.981+0000 7f0084d2a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:04.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.981+0000 7f0084d2a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:04.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.981+0000 7f0084d2a640 10 monclient: dump: 2024-03-25T19:37:04.982 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:04.982 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:04.982 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:04.983 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:04.983 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:04.983 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:04.983 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:04.983 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:04.983 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:04.983 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:04.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.981+0000 7f0087f90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:04.992 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.993+0000 7f0084d2a640 1 -- 172.21.15.44:0/968849561 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f007c0017f0 con 0x7f00800b4f40 2024-03-25T19:37:04.992 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.993+0000 7f0084d2a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:04.993 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.993+0000 7f0084d2a640 1 -- 172.21.15.44:0/968849561 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f007c001c00 con 0x7f00800b4f40 2024-03-25T19:37:04.993 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.993+0000 7f0087f90640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:04.993 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.993+0000 7f0087f90640 4 set_mon_vals no callback set 2024-03-25T19:37:04.993 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.993+0000 7f0084d2a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:04.993 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.993+0000 7f0084d2a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:04.993 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.993+0000 7f0084d2a640 10 monclient: dump: 2024-03-25T19:37:04.993 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:04.993 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:04.993 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:04.993 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:04.993 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:04.993 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:04.993 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:04.993 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:04.994 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:04.994 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:04.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.993+0000 7f0087f90640 10 monclient: shutdown 2024-03-25T19:37:04.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.993+0000 7f0087f90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:04.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.993+0000 7f0087f90640 1 -- 172.21.15.44:0/968849561 >> v1:172.21.15.111:6789/0 conn(0x7f00800b4f40 legacy=0x7f008013f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:04.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.994+0000 7f0087f90640 1 -- 172.21.15.44:0/968849561 shutdown_connections 2024-03-25T19:37:04.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.994+0000 7f0087f90640 1 -- 172.21.15.44:0/968849561 >> 172.21.15.44:0/968849561 conn(0x7f00800617a0 msgr2=0x7f0080061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:04.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.994+0000 7f0087f90640 1 -- 172.21.15.44:0/968849561 shutdown_connections 2024-03-25T19:37:04.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.994+0000 7f0087f90640 1 -- 172.21.15.44:0/968849561 wait complete. 2024-03-25T19:37:04.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.994+0000 7f0087f90640 10 monclient: build_initial_monmap 2024-03-25T19:37:04.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.994+0000 7f0087f90640 10 monclient: monmap: 2024-03-25T19:37:04.997 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:37:04.997 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:04.997 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:37:04.997 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:37:04.997 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:04.997 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:37:04.997 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:37:04.997 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:37:04.997 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:37:04.997 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:04.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.995+0000 7f0087f90640 1 librados: starting msgr at 2024-03-25T19:37:04.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.995+0000 7f0087f90640 1 librados: starting objecter 2024-03-25T19:37:04.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.995+0000 7f0087f90640 1 Processor -- start 2024-03-25T19:37:04.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.995+0000 7f0087f90640 1 -- start start 2024-03-25T19:37:04.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.995+0000 7f0087f90640 1 librados: setting wanted keys 2024-03-25T19:37:04.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.995+0000 7f0087f90640 1 librados: calling monclient init 2024-03-25T19:37:04.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.995+0000 7f0087f90640 10 monclient: init 2024-03-25T19:37:04.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.995+0000 7f0087f90640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:04.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.995+0000 7f0087f90640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:37:04.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.995+0000 7f0087f90640 10 monclient(hunting): picked mon.noname-b con 0x7f00800b4540 addr v1:172.21.15.111:6789/0 2024-03-25T19:37:04.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.995+0000 7f0087f90640 10 monclient(hunting): picked mon.noname-a con 0x7f00800b4f40 addr v1:172.21.15.44:6789/0 2024-03-25T19:37:04.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.996+0000 7f0087f90640 10 monclient(hunting): picked mon.noname-c con 0x7f008013fd90 addr v1:172.21.15.44:6790/0 2024-03-25T19:37:04.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.996+0000 7f0087f90640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00800b0180 con 0x7f00800b4f40 2024-03-25T19:37:04.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.996+0000 7f0087f90640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00800b1350 con 0x7f00800b4540 2024-03-25T19:37:04.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.996+0000 7f0087f90640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00800b1770 con 0x7f008013fd90 2024-03-25T19:37:04.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.996+0000 7f0087f90640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:04.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:04.996+0000 7f0087f90640 10 monclient(hunting): authenticate will time out at 5489.166967s 2024-03-25T19:37:05.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.012+0000 7f0085d2c640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f00800b4540 0x7f0080138410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:45674/0 (socket says 172.21.15.44:45674) 2024-03-25T19:37:05.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.012+0000 7f0085d2c640 1 -- 172.21.15.44:0/829493392 learned_addr learned my addr 172.21.15.44:0/829493392 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:37:05.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.012+0000 7f00767fc640 10 client.?.objecter ms_handle_connect 0x7f00800b4540 2024-03-25T19:37:05.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.012+0000 7f00767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:05.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.015+0000 7f00767fc640 10 client.?.objecter ms_handle_connect 0x7f008013fd90 2024-03-25T19:37:05.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.015+0000 7f00767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:05.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.015+0000 7f00767fc640 10 client.?.objecter ms_handle_connect 0x7f00800b4f40 2024-03-25T19:37:05.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.015+0000 7f00767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:05.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.025+0000 7f00767fc640 1 -- 172.21.15.44:0/829493392 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224079527 0 0) 0x7f00800b1350 con 0x7f00800b4540 2024-03-25T19:37:05.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.025+0000 7f00767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:05.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.025+0000 7f00767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:05.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.025+0000 7f00767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:05.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.025+0000 7f00767fc640 10 monclient(hunting): my global_id is 4297 2024-03-25T19:37:05.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.025+0000 7f00767fc640 1 -- 172.21.15.44:0/829493392 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0064004eb0 con 0x7f00800b4540 2024-03-25T19:37:05.028 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.028+0000 7f00767fc640 1 -- 172.21.15.44:0/829493392 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909405154 0 0) 0x7f00800b1770 con 0x7f008013fd90 2024-03-25T19:37:05.028 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.028+0000 7f00767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:05.028 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.028+0000 7f00767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:05.028 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.028+0000 7f00767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:05.028 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.028+0000 7f00767fc640 10 monclient(hunting): my global_id is 4280 2024-03-25T19:37:05.029 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.028+0000 7f00767fc640 1 -- 172.21.15.44:0/829493392 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00800b1350 con 0x7f008013fd90 2024-03-25T19:37:05.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.041+0000 7f00767fc640 1 -- 172.21.15.44:0/829493392 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636711569 0 0) 0x7f00800b0180 con 0x7f00800b4f40 2024-03-25T19:37:05.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.041+0000 7f00767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:05.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.041+0000 7f00767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:05.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.041+0000 7f00767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:05.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.041+0000 7f00767fc640 10 monclient(hunting): my global_id is 4311 2024-03-25T19:37:05.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.041+0000 7f00767fc640 1 -- 172.21.15.44:0/829493392 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00800b1770 con 0x7f00800b4f40 2024-03-25T19:37:05.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.042+0000 7f00767fc640 1 -- 172.21.15.44:0/829493392 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2606339694 0 0) 0x7f0064004eb0 con 0x7f00800b4540 2024-03-25T19:37:05.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.042+0000 7f00767fc640 1 -- 172.21.15.44:0/829493392 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00800b0180 con 0x7f00800b4540 2024-03-25T19:37:05.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.042+0000 7f00767fc640 1 -- 172.21.15.44:0/829493392 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f007c001c00 con 0x7f00800b4540 2024-03-25T19:37:05.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.042+0000 7f00767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:05.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.042+0000 7f00767fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:05.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.042+0000 7f00767fc640 10 monclient(hunting): dump: 2024-03-25T19:37:05.042 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:05.042 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:05.042 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:05.042 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:05.042 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:05.042 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:05.043 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:05.043 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:05.043 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:05.043 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:05.043 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.042+0000 7f00767fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:05.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.052+0000 7f00767fc640 1 -- 172.21.15.44:0/829493392 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 725122636 0 0) 0x7f00800b1770 con 0x7f00800b4f40 2024-03-25T19:37:05.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.052+0000 7f00767fc640 1 -- 172.21.15.44:0/829493392 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0064004eb0 con 0x7f00800b4f40 2024-03-25T19:37:05.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.052+0000 7f00767fc640 1 -- 172.21.15.44:0/829493392 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f00700018d0 con 0x7f00800b4f40 2024-03-25T19:37:05.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.052+0000 7f00767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:05.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.052+0000 7f00767fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:05.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.052+0000 7f00767fc640 10 monclient(hunting): dump: 2024-03-25T19:37:05.052 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:05.052 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:05.052 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:05.052 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:05.052 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:05.052 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:05.052 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:05.052 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:05.052 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:05.052 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:05.055 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.055+0000 7f00767fc640 1 -- 172.21.15.44:0/829493392 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4270381520 0 0) 0x7f00800b1350 con 0x7f008013fd90 2024-03-25T19:37:05.055 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.055+0000 7f00767fc640 1 -- 172.21.15.44:0/829493392 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00800b1770 con 0x7f008013fd90 2024-03-25T19:37:05.055 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.055+0000 7f00767fc640 1 -- 172.21.15.44:0/829493392 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f0078001bf0 con 0x7f008013fd90 2024-03-25T19:37:05.055 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.055+0000 7f00767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:05.055 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.055+0000 7f00767fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:05.055 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.055+0000 7f00767fc640 10 monclient(hunting): dump: 2024-03-25T19:37:05.055 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:05.056 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:05.056 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:05.056 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:05.056 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:05.056 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:05.056 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:05.056 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:05.056 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:05.056 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:05.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.057+0000 7f00767fc640 1 -- 172.21.15.44:0/829493392 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2973176756 0 0) 0x7f00800b0180 con 0x7f00800b4540 2024-03-25T19:37:05.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.057+0000 7f00767fc640 1 -- 172.21.15.44:0/829493392 >> v1:172.21.15.44:6790/0 conn(0x7f008013fd90 legacy=0x7f00801d8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:05.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.057+0000 7f00767fc640 1 -- 172.21.15.44:0/829493392 >> v1:172.21.15.44:6789/0 conn(0x7f00800b4f40 legacy=0x7f00800af590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:05.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.057+0000 7f00767fc640 10 monclient: _finish_hunting 0 2024-03-25T19:37:05.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.057+0000 7f00767fc640 1 monclient: found mon.b 2024-03-25T19:37:05.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.057+0000 7f00767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:05.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.057+0000 7f00767fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:05.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.057+0000 7f00767fc640 1 -- 172.21.15.44:0/829493392 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00800b1900 con 0x7f00800b4540 2024-03-25T19:37:05.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.057+0000 7f00767fc640 10 monclient: _finish_auth 0 2024-03-25T19:37:05.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.057+0000 7f00767fc640 10 monclient: _check_auth_tickets 2024-03-25T19:37:05.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.057+0000 7f00767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:05.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.057+0000 7f0087f90640 5 monclient: authenticate success, global_id 4297 2024-03-25T19:37:05.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.057+0000 7f0087f90640 10 monclient: _renew_subs 2024-03-25T19:37:05.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.057+0000 7f0087f90640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:05.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.057+0000 7f0087f90640 1 -- 172.21.15.44:0/829493392 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0080138b20 con 0x7f00800b4540 2024-03-25T19:37:05.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.057+0000 7f0087f90640 10 client.4297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:05.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.057+0000 7f0087f90640 10 monclient: _renew_subs 2024-03-25T19:37:05.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.057+0000 7f0087f90640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:05.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.057+0000 7f0087f90640 1 -- 172.21.15.44:0/829493392 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00800afe80 con 0x7f00800b4540 2024-03-25T19:37:05.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.057+0000 7f0087f90640 1 librados: init done 2024-03-25T19:37:05.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.059+0000 7f0087f90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:05.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.059+0000 7f0087f90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:05.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.059+0000 7f0087f90640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:05.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.059+0000 7f0087f90640 1 -- 172.21.15.44:0/829493392 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00800b11c0 con 0x7f00800b4540 2024-03-25T19:37:05.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.339+0000 7f00767fc640 1 -- 172.21.15.44:0/829493392 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f007c001230 con 0x7f00800b4540 2024-03-25T19:37:05.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.339+0000 7f00767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:05.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.339+0000 7f00767fc640 1 -- 172.21.15.44:0/829493392 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f007c003b00 con 0x7f00800b4540 2024-03-25T19:37:05.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.339+0000 7f00767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:05.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.339+0000 7f00767fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:05.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.339+0000 7f00767fc640 10 monclient: dump: 2024-03-25T19:37:05.340 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:05.340 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:05.340 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:05.340 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:05.340 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:05.340 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:05.340 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:05.340 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:05.340 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:05.340 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:05.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.339+0000 7f0077fff640 4 set_mon_vals no callback set 2024-03-25T19:37:05.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.340+0000 7f00767fc640 1 -- 172.21.15.44:0/829493392 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f007c003d20 con 0x7f00800b4540 2024-03-25T19:37:05.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.341+0000 7f008552b640 10 monclient: get_auth_request con 0x7f006406e0f0 auth_method 0 2024-03-25T19:37:05.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.341+0000 7f00767fc640 10 client.4297.objecter ms_handle_connect 0x7f006406e0f0 2024-03-25T19:37:05.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.345+0000 7f00767fc640 1 -- 172.21.15.44:0/829493392 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f007c054900 con 0x7f00800b4540 2024-03-25T19:37:05.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.345+0000 7f00767fc640 10 client.4297.objecter ms_dispatch 0x7f0080000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:05.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.345+0000 7f00767fc640 3 client.4297.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:05.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.345+0000 7f00767fc640 3 client.4297.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:05.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.345+0000 7f00767fc640 20 client.4297.objecter dump_active .. 0 homeless 2024-03-25T19:37:05.346 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.345+0000 7f00767fc640 1 -- 172.21.15.44:0/829493392 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f007c050410 con 0x7f00800b4540 2024-03-25T19:37:05.346 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.346+0000 7f00767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:05.346 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.346+0000 7f00767fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:05.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.602+0000 7f0087f90640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:05.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.602+0000 7f0087f90640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:05.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.602+0000 7f0087f90640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:05.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.602+0000 7f0087f90640 1 -- 172.21.15.44:0/829493392 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00800aabb0 con 0x7f00800b4540 2024-03-25T19:37:05.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.643+0000 7f00767fc640 1 -- 172.21.15.44:0/829493392 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f007c052340 con 0x7f00800b4540 2024-03-25T19:37:05.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.643+0000 7f00767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:05.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.643+0000 7f00767fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:05.643 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:37:05.643 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:05.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.645+0000 7f0087f90640 10 librados: watch_flush enter 2024-03-25T19:37:05.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.645+0000 7f0087f90640 10 librados: watch_flush exit 2024-03-25T19:37:05.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.645+0000 7f0087f90640 20 client.4297.objecter shutdown clearing up homeless session... 2024-03-25T19:37:05.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.645+0000 7f0087f90640 10 client.4297.objecter successfully canceled tick 2024-03-25T19:37:05.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.645+0000 7f0087f90640 1 -- 172.21.15.44:0/829493392 >> v1:172.21.15.111:6800/3883215793 conn(0x7f006406e0f0 legacy=0x7f00640705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:05.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.645+0000 7f0087f90640 10 monclient: shutdown 2024-03-25T19:37:05.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.645+0000 7f0087f90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:05.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.645+0000 7f0087f90640 1 -- 172.21.15.44:0/829493392 >> v1:172.21.15.111:6789/0 conn(0x7f00800b4540 legacy=0x7f0080138410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:05.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.645+0000 7f0087f90640 1 -- 172.21.15.44:0/829493392 shutdown_connections 2024-03-25T19:37:05.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.645+0000 7f0087f90640 1 -- 172.21.15.44:0/829493392 >> 172.21.15.44:0/829493392 conn(0x7f00800617a0 msgr2=0x7f0080139d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:05.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.646+0000 7f0087f90640 1 -- 172.21.15.44:0/829493392 shutdown_connections 2024-03-25T19:37:05.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.646+0000 7f0087f90640 1 -- 172.21.15.44:0/829493392 wait complete. 2024-03-25T19:37:05.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:05.646+0000 7f0087f90640 1 librados: shutdown 2024-03-25T19:37:05.659 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:12.463 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:12.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.565+0000 7fae7f1f6640 10 monclient: get_monmap_and_config 2024-03-25T19:37:12.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.565+0000 7fae7f1f6640 10 monclient: build_initial_monmap 2024-03-25T19:37:12.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.565+0000 7fae7f1f6640 10 monclient: monmap: 2024-03-25T19:37:12.565 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:37:12.565 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:12.565 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:37:12.566308+0000 2024-03-25T19:37:12.565 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:37:12.566308+0000 2024-03-25T19:37:12.565 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:12.565 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:12.565 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:37:12.565 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:37:12.565 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:37:12.565 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:12.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.565+0000 7fae7f1f6640 1 Processor -- start 2024-03-25T19:37:12.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.565+0000 7fae7f1f6640 1 -- start start 2024-03-25T19:37:12.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.565+0000 7fae7f1f6640 10 monclient: init 2024-03-25T19:37:12.567 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.566+0000 7fae7f1f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:12.567 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.566+0000 7fae7f1f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:37:12.567 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.566+0000 7fae7f1f6640 10 monclient(hunting): picked mon.noname-b con 0x7fae780b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T19:37:12.567 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.566+0000 7fae7f1f6640 10 monclient(hunting): picked mon.noname-a con 0x7fae780b1730 addr v1:172.21.15.44:6789/0 2024-03-25T19:37:12.567 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.566+0000 7fae7f1f6640 10 monclient(hunting): picked mon.noname-c con 0x7fae7814be90 addr v1:172.21.15.111:6789/0 2024-03-25T19:37:12.567 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.566+0000 7fae7f1f6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae780af940 con 0x7fae780b1730 2024-03-25T19:37:12.567 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.566+0000 7fae7f1f6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae780b0a70 con 0x7fae7814be90 2024-03-25T19:37:12.567 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.566+0000 7fae7f1f6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae780b0d50 con 0x7fae780b4d60 2024-03-25T19:37:12.567 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.566+0000 7fae7f1f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:12.567 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.566+0000 7fae7f1f6640 10 monclient(hunting): authenticate will time out at 5496.737433s 2024-03-25T19:37:12.567 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.567+0000 7fae7d793640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fae7814be90 0x7fae7814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:45236/0 (socket says 172.21.15.44:45236) 2024-03-25T19:37:12.567 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.567+0000 7fae7d793640 1 -- 172.21.15.44:0/3193854054 learned_addr learned my addr 172.21.15.44:0/3193854054 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:37:12.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.616+0000 7fae777fe640 1 -- 172.21.15.44:0/3193854054 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2838690222 0 0) 0x7fae780af940 con 0x7fae780b1730 2024-03-25T19:37:12.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.616+0000 7fae777fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:12.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.616+0000 7fae777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:12.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.616+0000 7fae777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:12.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.616+0000 7fae777fe640 10 monclient(hunting): my global_id is 4314 2024-03-25T19:37:12.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.616+0000 7fae777fe640 1 -- 172.21.15.44:0/3193854054 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae68003520 con 0x7fae780b1730 2024-03-25T19:37:12.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.623+0000 7fae777fe640 1 -- 172.21.15.44:0/3193854054 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 837362492 0 0) 0x7fae68003520 con 0x7fae780b1730 2024-03-25T19:37:12.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.623+0000 7fae777fe640 1 -- 172.21.15.44:0/3193854054 >> v1:172.21.15.44:6790/0 conn(0x7fae780b4d60 legacy=0x7fae7813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:12.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.623+0000 7fae777fe640 1 -- 172.21.15.44:0/3193854054 >> v1:172.21.15.111:6789/0 conn(0x7fae7814be90 legacy=0x7fae7814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:12.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.623+0000 7fae777fe640 10 monclient: _finish_hunting 0 2024-03-25T19:37:12.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.623+0000 7fae777fe640 1 monclient: found mon.noname-a 2024-03-25T19:37:12.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.623+0000 7fae777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:12.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.623+0000 7fae777fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:37:12.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.623+0000 7fae777fe640 1 -- 172.21.15.44:0/3193854054 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae780b1030 con 0x7fae780b1730 2024-03-25T19:37:12.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.623+0000 7fae777fe640 10 monclient: _finish_auth 0 2024-03-25T19:37:12.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.623+0000 7fae777fe640 10 monclient: _check_auth_tickets 2024-03-25T19:37:12.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.623+0000 7fae777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:12.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.623+0000 7fae7f1f6640 5 monclient: authenticate success, global_id 4314 2024-03-25T19:37:12.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.623+0000 7fae7f1f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:12.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.636+0000 7fae777fe640 1 -- 172.21.15.44:0/3193854054 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fae64001690 con 0x7fae780b1730 2024-03-25T19:37:12.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.636+0000 7fae777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:12.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.636+0000 7fae777fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:12.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.636+0000 7fae777fe640 10 monclient: dump: 2024-03-25T19:37:12.637 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:12.637 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:12.637 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:12.637 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:12.637 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:12.637 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:12.637 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:12.637 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:12.637 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:12.637 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:12.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.636+0000 7fae777fe640 1 -- 172.21.15.44:0/3193854054 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae64001920 con 0x7fae780b1730 2024-03-25T19:37:12.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.636+0000 7fae777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:12.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.636+0000 7fae777fe640 1 -- 172.21.15.44:0/3193854054 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fae64001c30 con 0x7fae780b1730 2024-03-25T19:37:12.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.636+0000 7fae777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:12.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.636+0000 7fae777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:12.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.636+0000 7fae777fe640 10 monclient: dump: 2024-03-25T19:37:12.638 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:12.638 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:12.638 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:12.638 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:12.638 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:12.638 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:12.638 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:12.638 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:12.638 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:12.638 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:12.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.636+0000 7fae7f1f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:12.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.636+0000 7fae7f1f6640 4 set_mon_vals no callback set 2024-03-25T19:37:12.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.637+0000 7fae7f1f6640 10 monclient: shutdown 2024-03-25T19:37:12.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.637+0000 7fae7f1f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:12.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.637+0000 7fae7f1f6640 1 -- 172.21.15.44:0/3193854054 >> v1:172.21.15.44:6789/0 conn(0x7fae780b1730 legacy=0x7fae780b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:12.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.637+0000 7fae7f1f6640 1 -- 172.21.15.44:0/3193854054 shutdown_connections 2024-03-25T19:37:12.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.637+0000 7fae7f1f6640 1 -- 172.21.15.44:0/3193854054 >> 172.21.15.44:0/3193854054 conn(0x7fae780617a0 msgr2=0x7fae78061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:12.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.637+0000 7fae7f1f6640 1 -- 172.21.15.44:0/3193854054 shutdown_connections 2024-03-25T19:37:12.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.637+0000 7fae7f1f6640 1 -- 172.21.15.44:0/3193854054 wait complete. 2024-03-25T19:37:12.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.638+0000 7fae7f1f6640 10 monclient: build_initial_monmap 2024-03-25T19:37:12.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.638+0000 7fae7f1f6640 10 monclient: monmap: 2024-03-25T19:37:12.641 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:37:12.641 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:12.641 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:37:12.641 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:37:12.641 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:12.641 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:37:12.641 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:37:12.641 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:37:12.641 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:37:12.641 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:12.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.638+0000 7fae7f1f6640 1 librados: starting msgr at 2024-03-25T19:37:12.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.638+0000 7fae7f1f6640 1 librados: starting objecter 2024-03-25T19:37:12.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.638+0000 7fae7f1f6640 1 Processor -- start 2024-03-25T19:37:12.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.638+0000 7fae7f1f6640 1 -- start start 2024-03-25T19:37:12.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.638+0000 7fae7f1f6640 1 librados: setting wanted keys 2024-03-25T19:37:12.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.638+0000 7fae7f1f6640 1 librados: calling monclient init 2024-03-25T19:37:12.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.638+0000 7fae7f1f6640 10 monclient: init 2024-03-25T19:37:12.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.639+0000 7fae7f1f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:12.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.639+0000 7fae7f1f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:37:12.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.639+0000 7fae7f1f6640 10 monclient(hunting): picked mon.noname-c con 0x7fae780b1730 addr v1:172.21.15.44:6790/0 2024-03-25T19:37:12.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.639+0000 7fae7f1f6640 10 monclient(hunting): picked mon.noname-a con 0x7fae780b4d60 addr v1:172.21.15.44:6789/0 2024-03-25T19:37:12.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.639+0000 7fae7f1f6640 10 monclient(hunting): picked mon.noname-b con 0x7fae7814be90 addr v1:172.21.15.111:6789/0 2024-03-25T19:37:12.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.639+0000 7fae7f1f6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae78140d90 con 0x7fae780b4d60 2024-03-25T19:37:12.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.639+0000 7fae7f1f6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae781384d0 con 0x7fae7814be90 2024-03-25T19:37:12.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.639+0000 7fae7f1f6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae78138660 con 0x7fae780b1730 2024-03-25T19:37:12.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.639+0000 7fae7f1f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:12.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.639+0000 7fae7f1f6640 10 monclient(hunting): authenticate will time out at 5496.810528s 2024-03-25T19:37:12.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.654+0000 7fae7d793640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fae7814be90 0x7fae781db7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:45276/0 (socket says 172.21.15.44:45276) 2024-03-25T19:37:12.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.654+0000 7fae7d793640 1 -- 172.21.15.44:0/440072038 learned_addr learned my addr 172.21.15.44:0/440072038 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:37:12.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.654+0000 7fae757fa640 10 client.?.objecter ms_handle_connect 0x7fae7814be90 2024-03-25T19:37:12.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.654+0000 7fae757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:12.662 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.662+0000 7fae757fa640 10 client.?.objecter ms_handle_connect 0x7fae780b1730 2024-03-25T19:37:12.662 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.662+0000 7fae757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:12.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.664+0000 7fae757fa640 10 client.?.objecter ms_handle_connect 0x7fae780b4d60 2024-03-25T19:37:12.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.664+0000 7fae757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:12.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.676+0000 7fae757fa640 1 -- 172.21.15.44:0/440072038 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4043824353 0 0) 0x7fae78140d90 con 0x7fae780b4d60 2024-03-25T19:37:12.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.676+0000 7fae757fa640 10 monclient(hunting): _negotiate 2024-03-25T19:37:12.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.676+0000 7fae757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:12.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.676+0000 7fae757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:12.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.676+0000 7fae757fa640 10 monclient(hunting): my global_id is 4317 2024-03-25T19:37:12.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.676+0000 7fae757fa640 1 -- 172.21.15.44:0/440072038 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae5c004eb0 con 0x7fae780b4d60 2024-03-25T19:37:12.733 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.733+0000 7fae757fa640 1 -- 172.21.15.44:0/440072038 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1241341386 0 0) 0x7fae5c004eb0 con 0x7fae780b4d60 2024-03-25T19:37:12.733 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.733+0000 7fae757fa640 1 -- 172.21.15.44:0/440072038 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae78140d90 con 0x7fae780b4d60 2024-03-25T19:37:12.733 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.733+0000 7fae757fa640 1 -- 172.21.15.44:0/440072038 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fae64001650 con 0x7fae780b4d60 2024-03-25T19:37:12.733 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.733+0000 7fae757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:12.733 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.733+0000 7fae757fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:12.733 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.733+0000 7fae757fa640 10 monclient(hunting): dump: 2024-03-25T19:37:12.733 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:12.734 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:12.734 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:12.734 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:12.734 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:12.734 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:12.734 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:12.734 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:12.734 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:12.734 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:12.734 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.733+0000 7fae757fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:12.742 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.742+0000 7fae757fa640 1 -- 172.21.15.44:0/440072038 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269668105 0 0) 0x7fae780b0d50 con 0x7fae780b1730 2024-03-25T19:37:12.742 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.742+0000 7fae757fa640 10 monclient(hunting): _negotiate 2024-03-25T19:37:12.742 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.742+0000 7fae757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:12.742 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.742+0000 7fae757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:12.742 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.742+0000 7fae757fa640 10 monclient(hunting): my global_id is 4286 2024-03-25T19:37:12.742 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.742+0000 7fae757fa640 1 -- 172.21.15.44:0/440072038 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae5c004eb0 con 0x7fae780b1730 2024-03-25T19:37:12.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.750+0000 7fae757fa640 1 -- 172.21.15.44:0/440072038 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2495751211 0 0) 0x7fae78140d90 con 0x7fae780b4d60 2024-03-25T19:37:12.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.750+0000 7fae757fa640 1 -- 172.21.15.44:0/440072038 >> v1:172.21.15.44:6790/0 conn(0x7fae780b1730 legacy=0x7fae78134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:12.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.750+0000 7fae757fa640 1 -- 172.21.15.44:0/440072038 >> v1:172.21.15.111:6789/0 conn(0x7fae7814be90 legacy=0x7fae781db7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:12.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.750+0000 7fae757fa640 10 monclient: _finish_hunting 0 2024-03-25T19:37:12.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.750+0000 7fae757fa640 1 monclient: found mon.a 2024-03-25T19:37:12.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.750+0000 7fae757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:12.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.750+0000 7fae757fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:37:12.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.750+0000 7fae757fa640 1 -- 172.21.15.44:0/440072038 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae78140380 con 0x7fae780b4d60 2024-03-25T19:37:12.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.750+0000 7fae757fa640 10 monclient: _finish_auth 0 2024-03-25T19:37:12.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.750+0000 7fae757fa640 10 monclient: _check_auth_tickets 2024-03-25T19:37:12.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.750+0000 7fae757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:12.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.751+0000 7fae7f1f6640 5 monclient: authenticate success, global_id 4317 2024-03-25T19:37:12.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.751+0000 7fae7f1f6640 10 monclient: _renew_subs 2024-03-25T19:37:12.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.751+0000 7fae7f1f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:37:12.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.751+0000 7fae7f1f6640 1 -- 172.21.15.44:0/440072038 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae78140570 con 0x7fae780b4d60 2024-03-25T19:37:12.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.751+0000 7fae7f1f6640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:12.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.751+0000 7fae7f1f6640 10 monclient: _renew_subs 2024-03-25T19:37:12.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.751+0000 7fae7f1f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:37:12.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.751+0000 7fae7f1f6640 1 -- 172.21.15.44:0/440072038 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae78138a50 con 0x7fae780b4d60 2024-03-25T19:37:12.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.751+0000 7fae7f1f6640 1 librados: init done 2024-03-25T19:37:12.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.752+0000 7fae7f1f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:12.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.752+0000 7fae7f1f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:12.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.752+0000 7fae7f1f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:37:12.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:12.752+0000 7fae7f1f6640 1 -- 172.21.15.44:0/440072038 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae781401f0 con 0x7fae780b4d60 2024-03-25T19:37:13.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.077+0000 7fae757fa640 1 -- 172.21.15.44:0/440072038 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae64003960 con 0x7fae780b4d60 2024-03-25T19:37:13.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.077+0000 7fae757fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:13.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.077+0000 7fae757fa640 1 -- 172.21.15.44:0/440072038 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fae64001250 con 0x7fae780b4d60 2024-03-25T19:37:13.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.077+0000 7fae757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:13.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.077+0000 7fae757fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:13.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.077+0000 7fae757fa640 10 monclient: dump: 2024-03-25T19:37:13.077 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:13.077 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:13.077 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:13.077 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:13.078 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:13.078 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:13.078 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:13.078 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:13.078 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:13.078 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:13.078 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.077+0000 7fae76ffd640 4 set_mon_vals no callback set 2024-03-25T19:37:13.079 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.078+0000 7fae757fa640 1 -- 172.21.15.44:0/440072038 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7fae64019b20 con 0x7fae780b4d60 2024-03-25T19:37:13.079 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.078+0000 7fae7cf92640 10 monclient: get_auth_request con 0x7fae5c06dd40 auth_method 0 2024-03-25T19:37:13.079 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.079+0000 7fae757fa640 10 client.4317.objecter ms_handle_connect 0x7fae5c06dd40 2024-03-25T19:37:13.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.083+0000 7fae757fa640 1 -- 172.21.15.44:0/440072038 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7fae64053800 con 0x7fae780b4d60 2024-03-25T19:37:13.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.083+0000 7fae757fa640 10 client.4317.objecter ms_dispatch 0x7fae78000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:13.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.084+0000 7fae757fa640 3 client.4317.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:13.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.084+0000 7fae757fa640 3 client.4317.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:13.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.084+0000 7fae757fa640 20 client.4317.objecter dump_active .. 0 homeless 2024-03-25T19:37:13.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.084+0000 7fae757fa640 1 -- 172.21.15.44:0/440072038 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fae6404f140 con 0x7fae780b4d60 2024-03-25T19:37:13.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.084+0000 7fae757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:13.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.084+0000 7fae757fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:13.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.340+0000 7fae7f1f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:13.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.340+0000 7fae7f1f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:13.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.340+0000 7fae7f1f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:37:13.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.340+0000 7fae7f1f6640 1 -- 172.21.15.44:0/440072038 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae780627b0 con 0x7fae780b4d60 2024-03-25T19:37:13.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.369+0000 7fae757fa640 1 -- 172.21.15.44:0/440072038 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7fae64051240 con 0x7fae780b4d60 2024-03-25T19:37:13.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.369+0000 7fae757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:13.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.369+0000 7fae757fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:13.369 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:37:13.369 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:13.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.371+0000 7fae7f1f6640 10 librados: watch_flush enter 2024-03-25T19:37:13.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.371+0000 7fae7f1f6640 10 librados: watch_flush exit 2024-03-25T19:37:13.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.371+0000 7fae7f1f6640 20 client.4317.objecter shutdown clearing up homeless session... 2024-03-25T19:37:13.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.371+0000 7fae7f1f6640 10 client.4317.objecter successfully canceled tick 2024-03-25T19:37:13.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.371+0000 7fae7f1f6640 1 -- 172.21.15.44:0/440072038 >> v1:172.21.15.111:6800/3883215793 conn(0x7fae5c06dd40 legacy=0x7fae5c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:13.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.371+0000 7fae7f1f6640 10 monclient: shutdown 2024-03-25T19:37:13.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.371+0000 7fae7f1f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:13.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.371+0000 7fae7f1f6640 1 -- 172.21.15.44:0/440072038 >> v1:172.21.15.44:6789/0 conn(0x7fae780b4d60 legacy=0x7fae781db0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:13.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.371+0000 7fae7f1f6640 1 -- 172.21.15.44:0/440072038 shutdown_connections 2024-03-25T19:37:13.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.371+0000 7fae7f1f6640 1 -- 172.21.15.44:0/440072038 >> 172.21.15.44:0/440072038 conn(0x7fae780617a0 msgr2=0x7fae780b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:13.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.371+0000 7fae7f1f6640 1 -- 172.21.15.44:0/440072038 shutdown_connections 2024-03-25T19:37:13.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.371+0000 7fae7f1f6640 1 -- 172.21.15.44:0/440072038 wait complete. 2024-03-25T19:37:13.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:13.371+0000 7fae7f1f6640 1 librados: shutdown 2024-03-25T19:37:13.383 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:20.189 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:20.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.305+0000 7f3ac5baf640 10 monclient: get_monmap_and_config 2024-03-25T19:37:20.306 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.306+0000 7f3ac5baf640 10 monclient: build_initial_monmap 2024-03-25T19:37:20.306 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.306+0000 7f3ac5baf640 10 monclient: monmap: 2024-03-25T19:37:20.306 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:37:20.306 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:20.306 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:37:20.307157+0000 2024-03-25T19:37:20.306 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:37:20.307157+0000 2024-03-25T19:37:20.306 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:20.306 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:20.306 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:37:20.306 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:37:20.306 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:37:20.307 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:20.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.306+0000 7f3ac5baf640 1 Processor -- start 2024-03-25T19:37:20.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.306+0000 7f3ac5baf640 1 -- start start 2024-03-25T19:37:20.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.306+0000 7f3ac5baf640 10 monclient: init 2024-03-25T19:37:20.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.307+0000 7f3ac5baf640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:20.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.307+0000 7f3ac5baf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:37:20.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.307+0000 7f3ac5baf640 10 monclient(hunting): picked mon.noname-b con 0x7f3ac00a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T19:37:20.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.307+0000 7f3ac5baf640 10 monclient(hunting): picked mon.noname-a con 0x7f3ac013f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:37:20.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.307+0000 7f3ac5baf640 10 monclient(hunting): picked mon.noname-c con 0x7f3ac01430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:37:20.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.307+0000 7f3ac5baf640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac0142520 con 0x7f3ac013f710 2024-03-25T19:37:20.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.307+0000 7f3ac5baf640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac00b28a0 con 0x7f3ac01430c0 2024-03-25T19:37:20.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.307+0000 7f3ac5baf640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac00b2a30 con 0x7f3ac00a98f0 2024-03-25T19:37:20.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.307+0000 7f3ac5baf640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:20.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.307+0000 7f3ac5baf640 10 monclient(hunting): authenticate will time out at 5504.478455s 2024-03-25T19:37:20.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.323+0000 7f3abf7fe640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f3ac00a98f0 0x7f3ac00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:56072/0 (socket says 172.21.15.44:56072) 2024-03-25T19:37:20.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.323+0000 7f3abf7fe640 1 -- 172.21.15.44:0/3880985446 learned_addr learned my addr 172.21.15.44:0/3880985446 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:37:20.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.344+0000 7f3abe7fc640 1 -- 172.21.15.44:0/3880985446 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1848395167 0 0) 0x7f3ac0142520 con 0x7f3ac013f710 2024-03-25T19:37:20.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.344+0000 7f3abe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:20.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.344+0000 7f3abe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:20.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.344+0000 7f3abe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:20.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.344+0000 7f3abe7fc640 10 monclient(hunting): my global_id is 4320 2024-03-25T19:37:20.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.344+0000 7f3abe7fc640 1 -- 172.21.15.44:0/3880985446 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aa8003520 con 0x7f3ac013f710 2024-03-25T19:37:20.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.344+0000 7f3abe7fc640 1 -- 172.21.15.44:0/3880985446 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436056484 0 0) 0x7f3ac00b2a30 con 0x7f3ac00a98f0 2024-03-25T19:37:20.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.344+0000 7f3abe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:20.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.344+0000 7f3abe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:20.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.344+0000 7f3abe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:20.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.344+0000 7f3abe7fc640 10 monclient(hunting): my global_id is 4289 2024-03-25T19:37:20.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.344+0000 7f3abe7fc640 1 -- 172.21.15.44:0/3880985446 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ac0142520 con 0x7f3ac00a98f0 2024-03-25T19:37:20.346 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.345+0000 7f3abe7fc640 1 -- 172.21.15.44:0/3880985446 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 340820725 0 0) 0x7f3ac00b28a0 con 0x7f3ac01430c0 2024-03-25T19:37:20.346 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.345+0000 7f3abe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:20.346 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.345+0000 7f3abe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:20.346 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.345+0000 7f3abe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:20.346 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.345+0000 7f3abe7fc640 10 monclient(hunting): my global_id is 4306 2024-03-25T19:37:20.346 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.345+0000 7f3abe7fc640 1 -- 172.21.15.44:0/3880985446 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ac00b2a30 con 0x7f3ac01430c0 2024-03-25T19:37:20.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.354+0000 7f3abe7fc640 1 -- 172.21.15.44:0/3880985446 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1776973976 0 0) 0x7f3ac00b2a30 con 0x7f3ac01430c0 2024-03-25T19:37:20.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.355+0000 7f3abe7fc640 1 -- 172.21.15.44:0/3880985446 >> v1:172.21.15.44:6790/0 conn(0x7f3ac00a98f0 legacy=0x7f3ac00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:20.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.355+0000 7f3abe7fc640 1 -- 172.21.15.44:0/3880985446 >> v1:172.21.15.44:6789/0 conn(0x7f3ac013f710 legacy=0x7f3ac01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:20.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.355+0000 7f3abe7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:37:20.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.355+0000 7f3abe7fc640 1 monclient: found mon.noname-c 2024-03-25T19:37:20.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.355+0000 7f3abe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:20.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.355+0000 7f3abe7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:37:20.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.355+0000 7f3abe7fc640 1 -- 172.21.15.44:0/3880985446 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac00b44c0 con 0x7f3ac01430c0 2024-03-25T19:37:20.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.355+0000 7f3abe7fc640 10 monclient: _finish_auth 0 2024-03-25T19:37:20.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.355+0000 7f3abe7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:37:20.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.355+0000 7f3abe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:20.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.355+0000 7f3abe7fc640 1 -- 172.21.15.44:0/3880985446 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3ab8001930 con 0x7f3ac01430c0 2024-03-25T19:37:20.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.355+0000 7f3abe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:20.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.355+0000 7f3abe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:20.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.355+0000 7f3abe7fc640 10 monclient: dump: 2024-03-25T19:37:20.356 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:20.356 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:20.356 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:20.356 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:20.356 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:20.356 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:20.357 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:20.357 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:20.357 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:20.357 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:20.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.355+0000 7f3ac5baf640 5 monclient: authenticate success, global_id 4306 2024-03-25T19:37:20.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.355+0000 7f3ac5baf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:20.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.367+0000 7f3abe7fc640 1 -- 172.21.15.44:0/3880985446 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ab8001a90 con 0x7f3ac01430c0 2024-03-25T19:37:20.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.367+0000 7f3abe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:20.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.367+0000 7f3abe7fc640 1 -- 172.21.15.44:0/3880985446 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3ab8001ee0 con 0x7f3ac01430c0 2024-03-25T19:37:20.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.367+0000 7f3ac5baf640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:20.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.367+0000 7f3ac5baf640 4 set_mon_vals no callback set 2024-03-25T19:37:20.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.367+0000 7f3abe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:20.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.367+0000 7f3abe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:20.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.367+0000 7f3abe7fc640 10 monclient: dump: 2024-03-25T19:37:20.367 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:20.368 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:20.368 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:20.368 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:20.368 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:20.368 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:20.368 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:20.368 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:20.368 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:20.368 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:20.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.368+0000 7f3ac5baf640 10 monclient: shutdown 2024-03-25T19:37:20.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.368+0000 7f3ac5baf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:20.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.368+0000 7f3ac5baf640 1 -- 172.21.15.44:0/3880985446 >> v1:172.21.15.111:6789/0 conn(0x7f3ac01430c0 legacy=0x7f3ac014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:20.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.368+0000 7f3ac5baf640 1 -- 172.21.15.44:0/3880985446 shutdown_connections 2024-03-25T19:37:20.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.368+0000 7f3ac5baf640 1 -- 172.21.15.44:0/3880985446 >> 172.21.15.44:0/3880985446 conn(0x7f3ac00b7ea0 msgr2=0x7f3ac00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:20.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.368+0000 7f3ac5baf640 1 -- 172.21.15.44:0/3880985446 shutdown_connections 2024-03-25T19:37:20.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.368+0000 7f3ac5baf640 1 -- 172.21.15.44:0/3880985446 wait complete. 2024-03-25T19:37:20.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.368+0000 7f3ac5baf640 10 monclient: build_initial_monmap 2024-03-25T19:37:20.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.369+0000 7f3ac5baf640 10 monclient: monmap: 2024-03-25T19:37:20.371 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:37:20.371 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:20.371 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:37:20.371 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:37:20.371 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:20.371 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:37:20.371 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:37:20.371 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:37:20.371 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:37:20.371 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:20.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.369+0000 7f3ac5baf640 1 librados: starting msgr at 2024-03-25T19:37:20.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.369+0000 7f3ac5baf640 1 librados: starting objecter 2024-03-25T19:37:20.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.369+0000 7f3ac5baf640 1 Processor -- start 2024-03-25T19:37:20.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.369+0000 7f3ac5baf640 1 -- start start 2024-03-25T19:37:20.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.369+0000 7f3ac5baf640 1 librados: setting wanted keys 2024-03-25T19:37:20.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.369+0000 7f3ac5baf640 1 librados: calling monclient init 2024-03-25T19:37:20.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.369+0000 7f3ac5baf640 10 monclient: init 2024-03-25T19:37:20.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.370+0000 7f3ac5baf640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:20.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.370+0000 7f3ac5baf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:37:20.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.370+0000 7f3ac5baf640 10 monclient(hunting): picked mon.noname-a con 0x7f3ac00a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:37:20.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.370+0000 7f3ac5baf640 10 monclient(hunting): picked mon.noname-c con 0x7f3ac013f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:37:20.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.370+0000 7f3ac5baf640 10 monclient(hunting): picked mon.noname-b con 0x7f3ac01430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:37:20.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.370+0000 7f3ac5baf640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac00aba90 con 0x7f3ac00a98f0 2024-03-25T19:37:20.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.370+0000 7f3ac5baf640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac01d7880 con 0x7f3ac01430c0 2024-03-25T19:37:20.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.370+0000 7f3ac5baf640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac01d84b0 con 0x7f3ac013f710 2024-03-25T19:37:20.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.370+0000 7f3ac5baf640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:20.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.370+0000 7f3ac5baf640 10 monclient(hunting): authenticate will time out at 5504.541378s 2024-03-25T19:37:20.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.378+0000 7f3abffff640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3ac01430c0 0x7f3ac0138160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:47966/0 (socket says 172.21.15.44:47966) 2024-03-25T19:37:20.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.378+0000 7f3abffff640 1 -- 172.21.15.44:0/198102713 learned_addr learned my addr 172.21.15.44:0/198102713 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:37:20.383 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.383+0000 7f3abf7fe640 1 -- 172.21.15.44:0/198102713 >> v1:172.21.15.44:6789/0 conn(0x7f3ac00a98f0 legacy=0x7f3ac01d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T19:37:20.383 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.383+0000 7f3abf7fe640 1 -- 172.21.15.44:0/198102713 >> v1:172.21.15.44:6789/0 conn(0x7f3ac00a98f0 legacy=0x7f3ac01d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:37:20.383 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.383+0000 7f3abf7fe640 1 --1- 172.21.15.44:0/198102713 >> v1:172.21.15.44:6789/0 conn(0x7f3ac00a98f0 0x7f3ac01d7da0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-03-25T19:37:20.383 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.383+0000 7f3abf7fe640 1 --1- 172.21.15.44:0/198102713 >> v1:172.21.15.44:6789/0 conn(0x7f3ac00a98f0 0x7f3ac01d7da0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-25T19:37:20.383 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.383+0000 7f3aa7fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.44:6789/0 2024-03-25T19:37:20.384 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.385+0000 7f3aa7fff640 10 client.?.objecter ms_handle_connect 0x7f3ac01430c0 2024-03-25T19:37:20.384 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.385+0000 7f3aa7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:20.388 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.389+0000 7f3aa7fff640 10 client.?.objecter ms_handle_connect 0x7f3ac013f710 2024-03-25T19:37:20.389 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.389+0000 7f3aa7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:20.403 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.403+0000 7f3aa7fff640 1 -- 172.21.15.44:0/198102713 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661521093 0 0) 0x7f3ac01d7880 con 0x7f3ac01430c0 2024-03-25T19:37:20.403 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.403+0000 7f3aa7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:20.403 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.403+0000 7f3aa7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:20.403 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.403+0000 7f3aa7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:20.403 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.403+0000 7f3aa7fff640 10 monclient(hunting): my global_id is 4309 2024-03-25T19:37:20.403 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.403+0000 7f3aa7fff640 1 -- 172.21.15.44:0/198102713 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aac004eb0 con 0x7f3ac01430c0 2024-03-25T19:37:20.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.405+0000 7f3aa7fff640 1 -- 172.21.15.44:0/198102713 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818138146 0 0) 0x7f3ac01d84b0 con 0x7f3ac013f710 2024-03-25T19:37:20.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.405+0000 7f3aa7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:20.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.405+0000 7f3aa7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:20.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.405+0000 7f3aa7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:20.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.405+0000 7f3aa7fff640 10 monclient(hunting): my global_id is 4292 2024-03-25T19:37:20.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.405+0000 7f3aa7fff640 1 -- 172.21.15.44:0/198102713 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ac01d7880 con 0x7f3ac013f710 2024-03-25T19:37:20.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.416+0000 7f3aa7fff640 1 -- 172.21.15.44:0/198102713 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2590235291 0 0) 0x7f3aac004eb0 con 0x7f3ac01430c0 2024-03-25T19:37:20.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.416+0000 7f3aa7fff640 1 -- 172.21.15.44:0/198102713 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ac01d84b0 con 0x7f3ac01430c0 2024-03-25T19:37:20.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.416+0000 7f3aa7fff640 1 -- 172.21.15.44:0/198102713 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3ab8001ea0 con 0x7f3ac01430c0 2024-03-25T19:37:20.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.416+0000 7f3aa7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:20.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.416+0000 7f3aa7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:20.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.416+0000 7f3aa7fff640 10 monclient(hunting): dump: 2024-03-25T19:37:20.417 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:20.417 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:20.417 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:20.417 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:20.417 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:20.417 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:20.417 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:20.417 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:20.417 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:20.417 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:20.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.416+0000 7f3aa7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:20.420 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.420+0000 7f3aa7fff640 1 -- 172.21.15.44:0/198102713 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 317526874 0 0) 0x7f3ac01d7880 con 0x7f3ac013f710 2024-03-25T19:37:20.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.421+0000 7f3aa7fff640 1 -- 172.21.15.44:0/198102713 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3aac004eb0 con 0x7f3ac013f710 2024-03-25T19:37:20.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.421+0000 7f3aa7fff640 1 -- 172.21.15.44:0/198102713 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3ab4001950 con 0x7f3ac013f710 2024-03-25T19:37:20.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.421+0000 7f3aa7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:20.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.421+0000 7f3aa7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:20.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.421+0000 7f3aa7fff640 10 monclient(hunting): dump: 2024-03-25T19:37:20.421 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:20.421 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:20.421 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:20.421 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:20.421 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:20.421 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:20.421 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:20.421 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:20.421 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:20.422 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:20.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.426+0000 7f3aa7fff640 1 -- 172.21.15.44:0/198102713 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3093646668 0 0) 0x7f3ac01d84b0 con 0x7f3ac01430c0 2024-03-25T19:37:20.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.426+0000 7f3aa7fff640 1 -- 172.21.15.44:0/198102713 >> v1:172.21.15.44:6790/0 conn(0x7f3ac013f710 legacy=0x7f3ac01d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:20.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.426+0000 7f3aa7fff640 1 -- 172.21.15.44:0/198102713 >> v1:172.21.15.44:6789/0 conn(0x7f3ac00a98f0 legacy=0x7f3ac01d7da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:37:20.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.426+0000 7f3aa7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:37:20.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.426+0000 7f3aa7fff640 1 monclient: found mon.b 2024-03-25T19:37:20.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.426+0000 7f3aa7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:20.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.426+0000 7f3aa7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:20.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.426+0000 7f3aa7fff640 1 -- 172.21.15.44:0/198102713 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac01d8640 con 0x7f3ac01430c0 2024-03-25T19:37:20.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.426+0000 7f3aa7fff640 10 monclient: _finish_auth 0 2024-03-25T19:37:20.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.426+0000 7f3aa7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:37:20.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.426+0000 7f3aa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:20.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.426+0000 7f3ac5baf640 5 monclient: authenticate success, global_id 4309 2024-03-25T19:37:20.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.426+0000 7f3ac5baf640 10 monclient: _renew_subs 2024-03-25T19:37:20.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.426+0000 7f3ac5baf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:20.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.426+0000 7f3ac5baf640 1 -- 172.21.15.44:0/198102713 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac01d41e0 con 0x7f3ac01430c0 2024-03-25T19:37:20.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.426+0000 7f3ac5baf640 10 client.4309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:20.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.426+0000 7f3ac5baf640 10 monclient: _renew_subs 2024-03-25T19:37:20.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.426+0000 7f3ac5baf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:20.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.426+0000 7f3ac5baf640 1 -- 172.21.15.44:0/198102713 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac01d5110 con 0x7f3ac01430c0 2024-03-25T19:37:20.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.426+0000 7f3ac5baf640 1 librados: init done 2024-03-25T19:37:20.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.427+0000 7f3ac5baf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:20.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.427+0000 7f3ac5baf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:20.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.427+0000 7f3ac5baf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:20.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.427+0000 7f3ac5baf640 1 -- 172.21.15.44:0/198102713 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac01d53c0 con 0x7f3ac01430c0 2024-03-25T19:37:20.685 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.685+0000 7f3aa7fff640 1 -- 172.21.15.44:0/198102713 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ab8003c20 con 0x7f3ac01430c0 2024-03-25T19:37:20.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.685+0000 7f3aa7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:20.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.685+0000 7f3aa7fff640 1 -- 172.21.15.44:0/198102713 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3ab8002100 con 0x7f3ac01430c0 2024-03-25T19:37:20.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.685+0000 7f3aa7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:20.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.685+0000 7f3aa7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:20.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.685+0000 7f3aa7fff640 10 monclient: dump: 2024-03-25T19:37:20.686 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:20.686 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:20.686 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:20.686 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:20.686 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:20.686 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:20.687 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:20.687 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:20.687 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:20.687 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:20.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.685+0000 7f3abdffb640 4 set_mon_vals no callback set 2024-03-25T19:37:20.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.686+0000 7f3aa7fff640 1 -- 172.21.15.44:0/198102713 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f3ab8004070 con 0x7f3ac01430c0 2024-03-25T19:37:20.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.687+0000 7f3abf7fe640 10 monclient: get_auth_request con 0x7f3aac06de90 auth_method 0 2024-03-25T19:37:20.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.687+0000 7f3aa7fff640 10 client.4309.objecter ms_handle_connect 0x7f3aac06de90 2024-03-25T19:37:20.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.691+0000 7f3aa7fff640 1 -- 172.21.15.44:0/198102713 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f3ab8054c90 con 0x7f3ac01430c0 2024-03-25T19:37:20.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.691+0000 7f3aa7fff640 10 client.4309.objecter ms_dispatch 0x7f3ac0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:20.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.691+0000 7f3aa7fff640 3 client.4309.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:20.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.691+0000 7f3aa7fff640 3 client.4309.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:20.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.691+0000 7f3aa7fff640 20 client.4309.objecter dump_active .. 0 homeless 2024-03-25T19:37:20.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.691+0000 7f3aa7fff640 1 -- 172.21.15.44:0/198102713 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3ab80507a0 con 0x7f3ac01430c0 2024-03-25T19:37:20.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.691+0000 7f3aa7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:20.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.691+0000 7f3aa7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:20.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.913+0000 7f3ac5baf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:20.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.913+0000 7f3ac5baf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:20.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.913+0000 7f3ac5baf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:20.913 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.913+0000 7f3ac5baf640 1 -- 172.21.15.44:0/198102713 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ac00b27a0 con 0x7f3ac01430c0 2024-03-25T19:37:20.947 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.947+0000 7f3aa7fff640 1 -- 172.21.15.44:0/198102713 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f3ab80526d0 con 0x7f3ac01430c0 2024-03-25T19:37:20.947 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.947+0000 7f3aa7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:20.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.947+0000 7f3aa7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:20.948 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:37:20.948 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:20.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.950+0000 7f3ac5baf640 10 librados: watch_flush enter 2024-03-25T19:37:20.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.950+0000 7f3ac5baf640 10 librados: watch_flush exit 2024-03-25T19:37:20.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.950+0000 7f3ac5baf640 20 client.4309.objecter shutdown clearing up homeless session... 2024-03-25T19:37:20.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.950+0000 7f3ac5baf640 10 client.4309.objecter successfully canceled tick 2024-03-25T19:37:20.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.950+0000 7f3ac5baf640 1 -- 172.21.15.44:0/198102713 >> v1:172.21.15.111:6800/3883215793 conn(0x7f3aac06de90 legacy=0x7f3aac070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:20.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.950+0000 7f3ac5baf640 10 monclient: shutdown 2024-03-25T19:37:20.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.950+0000 7f3ac5baf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:20.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.950+0000 7f3ac5baf640 1 -- 172.21.15.44:0/198102713 >> v1:172.21.15.111:6789/0 conn(0x7f3ac01430c0 legacy=0x7f3ac0138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:20.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.950+0000 7f3ac5baf640 1 -- 172.21.15.44:0/198102713 shutdown_connections 2024-03-25T19:37:20.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.950+0000 7f3ac5baf640 1 -- 172.21.15.44:0/198102713 >> 172.21.15.44:0/198102713 conn(0x7f3ac00b7ea0 msgr2=0x7f3ac0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:20.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.950+0000 7f3ac5baf640 1 -- 172.21.15.44:0/198102713 shutdown_connections 2024-03-25T19:37:20.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.950+0000 7f3ac5baf640 1 -- 172.21.15.44:0/198102713 wait complete. 2024-03-25T19:37:20.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:20.950+0000 7f3ac5baf640 1 librados: shutdown 2024-03-25T19:37:20.964 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:23.759 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:02:14.661 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:23.810 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:02:14.712 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:23.857 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:02:14.737 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:23.862 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:02:14.764 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:23.865 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:02:14.745 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:23.869 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:02:14.771 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:23.875 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:02:14.777 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:23.890 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:02:14.770 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:23.907 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:02:14.809 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:27.770 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:27.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.885+0000 7f3fdfa06640 10 monclient: get_monmap_and_config 2024-03-25T19:37:27.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.885+0000 7f3fdfa06640 10 monclient: build_initial_monmap 2024-03-25T19:37:27.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.885+0000 7f3fdfa06640 10 monclient: monmap: 2024-03-25T19:37:27.886 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:37:27.886 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:27.886 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:37:27.886881+0000 2024-03-25T19:37:27.886 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:37:27.886881+0000 2024-03-25T19:37:27.886 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:27.886 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:27.886 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:37:27.886 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:37:27.886 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:37:27.886 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:27.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.886+0000 7f3fdfa06640 1 Processor -- start 2024-03-25T19:37:27.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.886+0000 7f3fdfa06640 1 -- start start 2024-03-25T19:37:27.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.886+0000 7f3fdfa06640 10 monclient: init 2024-03-25T19:37:27.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.887+0000 7f3fdfa06640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:27.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.887+0000 7f3fdfa06640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:37:27.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.887+0000 7f3fdfa06640 10 monclient(hunting): picked mon.noname-a con 0x7f3fd80b10d0 addr v1:172.21.15.44:6789/0 2024-03-25T19:37:27.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.887+0000 7f3fdfa06640 10 monclient(hunting): picked mon.noname-c con 0x7f3fd80b4360 addr v1:172.21.15.111:6789/0 2024-03-25T19:37:27.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.887+0000 7f3fdfa06640 10 monclient(hunting): picked mon.noname-b con 0x7f3fd80a9470 addr v1:172.21.15.44:6790/0 2024-03-25T19:37:27.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.887+0000 7f3fdfa06640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fd80a9ab0 con 0x7f3fd80b10d0 2024-03-25T19:37:27.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.887+0000 7f3fdfa06640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fd80b2400 con 0x7f3fd80b4360 2024-03-25T19:37:27.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.887+0000 7f3fdfa06640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fd80aa2a0 con 0x7f3fd80a9470 2024-03-25T19:37:27.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.887+0000 7f3fdfa06640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:27.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.887+0000 7f3fdfa06640 10 monclient(hunting): authenticate will time out at 5512.058659s 2024-03-25T19:37:27.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.893+0000 7f3fdd7a2640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f3fd80b10d0 0x7f3fd80b4ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40344/0 (socket says 172.21.15.44:40344) 2024-03-25T19:37:27.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.894+0000 7f3fdd7a2640 1 -- 172.21.15.44:0/3525121812 learned_addr learned my addr 172.21.15.44:0/3525121812 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:37:27.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.909+0000 7f3fcffff640 1 -- 172.21.15.44:0/3525121812 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754937622 0 0) 0x7f3fd80b2400 con 0x7f3fd80b4360 2024-03-25T19:37:27.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.909+0000 7f3fcffff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:27.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.909+0000 7f3fcffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:27.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.909+0000 7f3fcffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:27.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.909+0000 7f3fcffff640 10 monclient(hunting): my global_id is 4312 2024-03-25T19:37:27.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.909+0000 7f3fcffff640 1 -- 172.21.15.44:0/3525121812 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fc4003520 con 0x7f3fd80b4360 2024-03-25T19:37:27.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.915+0000 7f3fcffff640 1 -- 172.21.15.44:0/3525121812 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950304325 0 0) 0x7f3fd80a9ab0 con 0x7f3fd80b10d0 2024-03-25T19:37:27.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.916+0000 7f3fcffff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:27.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.916+0000 7f3fcffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:27.915 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.916+0000 7f3fcffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:27.916 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.916+0000 7f3fcffff640 10 monclient(hunting): my global_id is 4323 2024-03-25T19:37:27.916 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.916+0000 7f3fcffff640 1 -- 172.21.15.44:0/3525121812 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fd80b2400 con 0x7f3fd80b10d0 2024-03-25T19:37:27.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.924+0000 7f3fcffff640 1 -- 172.21.15.44:0/3525121812 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2444402044 0 0) 0x7f3fc4003520 con 0x7f3fd80b4360 2024-03-25T19:37:27.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.924+0000 7f3fcffff640 1 -- 172.21.15.44:0/3525121812 >> v1:172.21.15.44:6790/0 conn(0x7f3fd80a9470 legacy=0x7f3fd814dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:27.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.924+0000 7f3fcffff640 1 -- 172.21.15.44:0/3525121812 >> v1:172.21.15.44:6789/0 conn(0x7f3fd80b10d0 legacy=0x7f3fd80b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:27.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.924+0000 7f3fcffff640 10 monclient: _finish_hunting 0 2024-03-25T19:37:27.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.924+0000 7f3fcffff640 1 monclient: found mon.noname-c 2024-03-25T19:37:27.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.924+0000 7f3fcffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:27.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.924+0000 7f3fcffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:37:27.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.924+0000 7f3fcffff640 1 -- 172.21.15.44:0/3525121812 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd80b14b0 con 0x7f3fd80b4360 2024-03-25T19:37:27.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.924+0000 7f3fcffff640 10 monclient: _finish_auth 0 2024-03-25T19:37:27.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.924+0000 7f3fcffff640 10 monclient: _check_auth_tickets 2024-03-25T19:37:27.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.924+0000 7f3fcffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:27.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.924+0000 7f3fcffff640 1 -- 172.21.15.44:0/3525121812 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3fc8001690 con 0x7f3fd80b4360 2024-03-25T19:37:27.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.924+0000 7f3fcffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:27.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.924+0000 7f3fcffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:27.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.924+0000 7f3fcffff640 10 monclient: dump: 2024-03-25T19:37:27.926 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:27.926 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:27.926 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:27.926 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:27.926 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:27.926 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:27.926 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:27.926 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:27.926 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:27.926 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:27.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.924+0000 7f3fdfa06640 5 monclient: authenticate success, global_id 4312 2024-03-25T19:37:27.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.924+0000 7f3fdfa06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:27.947 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.947+0000 7f3fcffff640 1 -- 172.21.15.44:0/3525121812 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3fc80017f0 con 0x7f3fd80b4360 2024-03-25T19:37:27.947 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.947+0000 7f3fcffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:27.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.947+0000 7f3fcffff640 1 -- 172.21.15.44:0/3525121812 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3fc8001c00 con 0x7f3fd80b4360 2024-03-25T19:37:27.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.947+0000 7f3fdfa06640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:27.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.947+0000 7f3fdfa06640 4 set_mon_vals no callback set 2024-03-25T19:37:27.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.947+0000 7f3fcffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:27.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.947+0000 7f3fcffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:27.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.947+0000 7f3fcffff640 10 monclient: dump: 2024-03-25T19:37:27.948 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:27.948 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:27.948 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:27.948 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:27.948 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:27.948 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:27.948 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:27.948 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:27.949 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:27.949 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:27.950 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.948+0000 7f3fdfa06640 10 monclient: shutdown 2024-03-25T19:37:27.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.948+0000 7f3fdfa06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:27.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.948+0000 7f3fdfa06640 1 -- 172.21.15.44:0/3525121812 >> v1:172.21.15.111:6789/0 conn(0x7f3fd80b4360 legacy=0x7f3fd80b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:27.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.948+0000 7f3fdfa06640 1 -- 172.21.15.44:0/3525121812 shutdown_connections 2024-03-25T19:37:27.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.948+0000 7f3fdfa06640 1 -- 172.21.15.44:0/3525121812 >> 172.21.15.44:0/3525121812 conn(0x7f3fd80617a0 msgr2=0x7f3fd8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:27.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.948+0000 7f3fdfa06640 1 -- 172.21.15.44:0/3525121812 shutdown_connections 2024-03-25T19:37:27.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.948+0000 7f3fdfa06640 1 -- 172.21.15.44:0/3525121812 wait complete. 2024-03-25T19:37:27.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.949+0000 7f3fdfa06640 10 monclient: build_initial_monmap 2024-03-25T19:37:27.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.949+0000 7f3fdfa06640 10 monclient: monmap: 2024-03-25T19:37:27.951 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:37:27.951 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:27.951 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:37:27.951 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:37:27.951 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:27.952 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:37:27.952 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:37:27.952 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:37:27.952 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:37:27.952 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:27.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.949+0000 7f3fdfa06640 1 librados: starting msgr at 2024-03-25T19:37:27.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.949+0000 7f3fdfa06640 1 librados: starting objecter 2024-03-25T19:37:27.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.950+0000 7f3fdfa06640 1 Processor -- start 2024-03-25T19:37:27.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.950+0000 7f3fdfa06640 1 -- start start 2024-03-25T19:37:27.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.950+0000 7f3fdfa06640 1 librados: setting wanted keys 2024-03-25T19:37:27.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.950+0000 7f3fdfa06640 1 librados: calling monclient init 2024-03-25T19:37:27.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.950+0000 7f3fdfa06640 10 monclient: init 2024-03-25T19:37:27.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.950+0000 7f3fdfa06640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:27.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.950+0000 7f3fdfa06640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:37:27.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.950+0000 7f3fdfa06640 10 monclient(hunting): picked mon.noname-c con 0x7f3fd80a9470 addr v1:172.21.15.44:6790/0 2024-03-25T19:37:27.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.950+0000 7f3fdfa06640 10 monclient(hunting): picked mon.noname-a con 0x7f3fd80b10d0 addr v1:172.21.15.44:6789/0 2024-03-25T19:37:27.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.950+0000 7f3fdfa06640 10 monclient(hunting): picked mon.noname-b con 0x7f3fd80b4360 addr v1:172.21.15.111:6789/0 2024-03-25T19:37:27.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.950+0000 7f3fdfa06640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fd81d7380 con 0x7f3fd80b10d0 2024-03-25T19:37:27.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.950+0000 7f3fdfa06640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fd80adba0 con 0x7f3fd80b4360 2024-03-25T19:37:27.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.950+0000 7f3fdfa06640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fd80b06f0 con 0x7f3fd80a9470 2024-03-25T19:37:27.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.950+0000 7f3fdfa06640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:27.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.950+0000 7f3fdfa06640 10 monclient(hunting): authenticate will time out at 5512.121716s 2024-03-25T19:37:27.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.960+0000 7f3fddfa3640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3fd80b4360 0x7f3fd81d7740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:52772/0 (socket says 172.21.15.44:52772) 2024-03-25T19:37:27.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.960+0000 7f3fddfa3640 1 -- 172.21.15.44:0/2574660058 learned_addr learned my addr 172.21.15.44:0/2574660058 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:37:27.960 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.960+0000 7f3fcdffb640 10 client.?.objecter ms_handle_connect 0x7f3fd80b4360 2024-03-25T19:37:27.960 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.960+0000 7f3fcdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:27.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.974+0000 7f3fcdffb640 1 -- 172.21.15.44:0/2574660058 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046865964 0 0) 0x7f3fd80adba0 con 0x7f3fd80b4360 2024-03-25T19:37:27.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.974+0000 7f3fcdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:27.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.974+0000 7f3fcdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:27.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.974+0000 7f3fcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:27.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.974+0000 7f3fcdffb640 10 monclient(hunting): my global_id is 4315 2024-03-25T19:37:27.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.974+0000 7f3fcdffb640 1 -- 172.21.15.44:0/2574660058 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fbc004eb0 con 0x7f3fd80b4360 2024-03-25T19:37:27.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.975+0000 7f3fcdffb640 10 client.?.objecter ms_handle_connect 0x7f3fd80a9470 2024-03-25T19:37:27.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.975+0000 7f3fcdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:27.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.981+0000 7f3fcdffb640 10 client.?.objecter ms_handle_connect 0x7f3fd80b10d0 2024-03-25T19:37:27.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.981+0000 7f3fcdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:27.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.983+0000 7f3fcdffb640 1 -- 172.21.15.44:0/2574660058 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 898433419 0 0) 0x7f3fd80b06f0 con 0x7f3fd80a9470 2024-03-25T19:37:27.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.983+0000 7f3fcdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:27.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.983+0000 7f3fcdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:27.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.983+0000 7f3fcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:27.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.983+0000 7f3fcdffb640 10 monclient(hunting): my global_id is 4298 2024-03-25T19:37:27.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.983+0000 7f3fcdffb640 1 -- 172.21.15.44:0/2574660058 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fd80adba0 con 0x7f3fd80a9470 2024-03-25T19:37:27.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.994+0000 7f3fcdffb640 1 -- 172.21.15.44:0/2574660058 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2706731587 0 0) 0x7f3fbc004eb0 con 0x7f3fd80b4360 2024-03-25T19:37:27.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.994+0000 7f3fcdffb640 1 -- 172.21.15.44:0/2574660058 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3fd80b06f0 con 0x7f3fd80b4360 2024-03-25T19:37:27.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.994+0000 7f3fcdffb640 1 -- 172.21.15.44:0/2574660058 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3fd0001d30 con 0x7f3fd80b4360 2024-03-25T19:37:27.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.994+0000 7f3fcdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:27.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.994+0000 7f3fcdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:27.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.994+0000 7f3fcdffb640 10 monclient(hunting): dump: 2024-03-25T19:37:27.994 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:27.994 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:27.994 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:27.994 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:27.994 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:27.995 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:27.995 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:27.995 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:27.995 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:27.995 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:27.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:27.994+0000 7f3fcdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:28.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.002+0000 7f3fcdffb640 1 -- 172.21.15.44:0/2574660058 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 976628469 0 0) 0x7f3fd81d7380 con 0x7f3fd80b10d0 2024-03-25T19:37:28.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.002+0000 7f3fcdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:28.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.002+0000 7f3fcdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:28.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.002+0000 7f3fcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:28.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.002+0000 7f3fcdffb640 10 monclient(hunting): my global_id is 4326 2024-03-25T19:37:28.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.002+0000 7f3fcdffb640 1 -- 172.21.15.44:0/2574660058 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fbc004eb0 con 0x7f3fd80b10d0 2024-03-25T19:37:28.007 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.007+0000 7f3fcdffb640 1 -- 172.21.15.44:0/2574660058 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 411484973 0 0) 0x7f3fd80adba0 con 0x7f3fd80a9470 2024-03-25T19:37:28.007 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.007+0000 7f3fcdffb640 1 -- 172.21.15.44:0/2574660058 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3fd81d7380 con 0x7f3fd80a9470 2024-03-25T19:37:28.008 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.007+0000 7f3fcdffb640 1 -- 172.21.15.44:0/2574660058 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3fd40018b0 con 0x7f3fd80a9470 2024-03-25T19:37:28.008 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.007+0000 7f3fcdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:28.008 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.007+0000 7f3fcdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:28.008 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.007+0000 7f3fcdffb640 10 monclient(hunting): dump: 2024-03-25T19:37:28.008 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:28.008 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:28.008 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:28.008 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:28.008 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:28.008 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:28.008 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:28.008 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:28.008 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:28.008 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:28.019 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.019+0000 7f3fcdffb640 1 -- 172.21.15.44:0/2574660058 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1056630788 0 0) 0x7f3fd80b06f0 con 0x7f3fd80b4360 2024-03-25T19:37:28.019 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.019+0000 7f3fcdffb640 1 -- 172.21.15.44:0/2574660058 >> v1:172.21.15.44:6790/0 conn(0x7f3fd80a9470 legacy=0x7f3fd80ab9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:28.019 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.019+0000 7f3fcdffb640 1 -- 172.21.15.44:0/2574660058 >> v1:172.21.15.44:6789/0 conn(0x7f3fd80b10d0 legacy=0x7f3fd81dac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:28.019 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.019+0000 7f3fcdffb640 10 monclient: _finish_hunting 0 2024-03-25T19:37:28.019 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.019+0000 7f3fcdffb640 1 monclient: found mon.b 2024-03-25T19:37:28.019 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.019+0000 7f3fcdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:28.019 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.019+0000 7f3fcdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:28.020 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.019+0000 7f3fcdffb640 1 -- 172.21.15.44:0/2574660058 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd813f8b0 con 0x7f3fd80b4360 2024-03-25T19:37:28.020 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.019+0000 7f3fcdffb640 10 monclient: _finish_auth 0 2024-03-25T19:37:28.020 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.019+0000 7f3fcdffb640 10 monclient: _check_auth_tickets 2024-03-25T19:37:28.020 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.019+0000 7f3fcdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:28.020 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.019+0000 7f3fdfa06640 5 monclient: authenticate success, global_id 4315 2024-03-25T19:37:28.020 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.019+0000 7f3fdfa06640 10 monclient: _renew_subs 2024-03-25T19:37:28.020 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.019+0000 7f3fdfa06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:28.020 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.019+0000 7f3fdfa06640 1 -- 172.21.15.44:0/2574660058 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fd81db370 con 0x7f3fd80b4360 2024-03-25T19:37:28.020 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.019+0000 7f3fdfa06640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:28.020 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.019+0000 7f3fdfa06640 10 monclient: _renew_subs 2024-03-25T19:37:28.020 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.019+0000 7f3fdfa06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:28.020 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.019+0000 7f3fdfa06640 1 -- 172.21.15.44:0/2574660058 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fd81d7e50 con 0x7f3fd80b4360 2024-03-25T19:37:28.020 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.019+0000 7f3fdfa06640 1 librados: init done 2024-03-25T19:37:28.021 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.020+0000 7f3fdfa06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:28.021 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.020+0000 7f3fdfa06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:28.021 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.020+0000 7f3fdfa06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:28.021 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.020+0000 7f3fdfa06640 1 -- 172.21.15.44:0/2574660058 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fd80ad9c0 con 0x7f3fd80b4360 2024-03-25T19:37:28.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.295+0000 7f3fcdffb640 1 -- 172.21.15.44:0/2574660058 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3fd0003990 con 0x7f3fd80b4360 2024-03-25T19:37:28.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.295+0000 7f3fcdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:28.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.295+0000 7f3fcdffb640 1 -- 172.21.15.44:0/2574660058 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3fd0003d20 con 0x7f3fd80b4360 2024-03-25T19:37:28.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.295+0000 7f3fcdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:28.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.295+0000 7f3fcdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:28.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.295+0000 7f3fcdffb640 10 monclient: dump: 2024-03-25T19:37:28.295 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:28.295 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:28.295 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:28.296 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:28.296 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:28.296 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:28.296 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:28.296 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:28.296 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:28.296 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:28.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.295+0000 7f3fcf7fe640 4 set_mon_vals no callback set 2024-03-25T19:37:28.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.296+0000 7f3fcdffb640 1 -- 172.21.15.44:0/2574660058 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f3fd0003f00 con 0x7f3fd80b4360 2024-03-25T19:37:28.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.296+0000 7f3fdd7a2640 10 monclient: get_auth_request con 0x7f3fbc06df70 auth_method 0 2024-03-25T19:37:28.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.297+0000 7f3fcdffb640 10 client.4315.objecter ms_handle_connect 0x7f3fbc06df70 2024-03-25T19:37:28.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.303+0000 7f3fcdffb640 1 -- 172.21.15.44:0/2574660058 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f3fd0054c00 con 0x7f3fd80b4360 2024-03-25T19:37:28.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.303+0000 7f3fcdffb640 10 client.4315.objecter ms_dispatch 0x7f3fd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:28.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.303+0000 7f3fcdffb640 3 client.4315.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:28.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.303+0000 7f3fcdffb640 3 client.4315.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:28.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.303+0000 7f3fcdffb640 20 client.4315.objecter dump_active .. 0 homeless 2024-03-25T19:37:28.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.303+0000 7f3fcdffb640 1 -- 172.21.15.44:0/2574660058 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3fd0050710 con 0x7f3fd80b4360 2024-03-25T19:37:28.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.303+0000 7f3fcdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:28.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.303+0000 7f3fcdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:28.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.527+0000 7f3fdfa06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:28.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.527+0000 7f3fdfa06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:28.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.527+0000 7f3fdfa06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:28.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.527+0000 7f3fdfa06640 1 -- 172.21.15.44:0/2574660058 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fd813e710 con 0x7f3fd80b4360 2024-03-25T19:37:28.571 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.571+0000 7f3fcdffb640 1 -- 172.21.15.44:0/2574660058 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f3fd0052640 con 0x7f3fd80b4360 2024-03-25T19:37:28.571 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.571+0000 7f3fcdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:28.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.571+0000 7f3fcdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:28.572 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:37:28.572 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:28.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.574+0000 7f3fdfa06640 10 librados: watch_flush enter 2024-03-25T19:37:28.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.574+0000 7f3fdfa06640 10 librados: watch_flush exit 2024-03-25T19:37:28.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.574+0000 7f3fdfa06640 20 client.4315.objecter shutdown clearing up homeless session... 2024-03-25T19:37:28.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.574+0000 7f3fdfa06640 10 client.4315.objecter successfully canceled tick 2024-03-25T19:37:28.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.574+0000 7f3fdfa06640 1 -- 172.21.15.44:0/2574660058 >> v1:172.21.15.111:6800/3883215793 conn(0x7f3fbc06df70 legacy=0x7f3fbc070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:28.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.574+0000 7f3fdfa06640 10 monclient: shutdown 2024-03-25T19:37:28.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.574+0000 7f3fdfa06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:28.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.574+0000 7f3fdfa06640 1 -- 172.21.15.44:0/2574660058 >> v1:172.21.15.111:6789/0 conn(0x7f3fd80b4360 legacy=0x7f3fd81d7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:28.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.574+0000 7f3fdfa06640 1 -- 172.21.15.44:0/2574660058 shutdown_connections 2024-03-25T19:37:28.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.574+0000 7f3fdfa06640 1 -- 172.21.15.44:0/2574660058 >> 172.21.15.44:0/2574660058 conn(0x7f3fd80617a0 msgr2=0x7f3fd8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:28.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.574+0000 7f3fdfa06640 1 -- 172.21.15.44:0/2574660058 shutdown_connections 2024-03-25T19:37:28.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.574+0000 7f3fdfa06640 1 -- 172.21.15.44:0/2574660058 wait complete. 2024-03-25T19:37:28.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:28.574+0000 7f3fdfa06640 1 librados: shutdown 2024-03-25T19:37:28.586 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:35.392 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:35.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.503+0000 7f3d2973d640 10 monclient: get_monmap_and_config 2024-03-25T19:37:35.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.503+0000 7f3d2973d640 10 monclient: build_initial_monmap 2024-03-25T19:37:35.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.503+0000 7f3d2973d640 10 monclient: monmap: 2024-03-25T19:37:35.503 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:37:35.503 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:35.503 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:37:35.504349+0000 2024-03-25T19:37:35.503 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:37:35.504349+0000 2024-03-25T19:37:35.503 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:35.503 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:35.504 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:37:35.504 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:37:35.504 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:37:35.504 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:35.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.504+0000 7f3d2973d640 1 Processor -- start 2024-03-25T19:37:35.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.504+0000 7f3d2973d640 1 -- start start 2024-03-25T19:37:35.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.504+0000 7f3d2973d640 10 monclient: init 2024-03-25T19:37:35.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.504+0000 7f3d2973d640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:35.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.504+0000 7f3d2973d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:37:35.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.504+0000 7f3d2973d640 10 monclient(hunting): picked mon.noname-a con 0x7f3d240a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:37:35.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.504+0000 7f3d2973d640 10 monclient(hunting): picked mon.noname-c con 0x7f3d2413f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:37:35.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.504+0000 7f3d2973d640 10 monclient(hunting): picked mon.noname-b con 0x7f3d241430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:37:35.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.504+0000 7f3d2973d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d24142520 con 0x7f3d240a98f0 2024-03-25T19:37:35.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.504+0000 7f3d2973d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d240b28a0 con 0x7f3d2413f710 2024-03-25T19:37:35.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.504+0000 7f3d2973d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d240b2a30 con 0x7f3d241430c0 2024-03-25T19:37:35.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.504+0000 7f3d2973d640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:35.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.504+0000 7f3d2973d640 10 monclient(hunting): authenticate will time out at 5519.675825s 2024-03-25T19:37:35.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.521+0000 7f3d237fe640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f3d241430c0 0x7f3d2414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:49316/0 (socket says 172.21.15.44:49316) 2024-03-25T19:37:35.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.521+0000 7f3d237fe640 1 -- 172.21.15.44:0/2367922705 learned_addr learned my addr 172.21.15.44:0/2367922705 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:37:35.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.547+0000 7f3d227fc640 1 -- 172.21.15.44:0/2367922705 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167906242 0 0) 0x7f3d24142520 con 0x7f3d240a98f0 2024-03-25T19:37:35.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.547+0000 7f3d227fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:35.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.547+0000 7f3d227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:35.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.547+0000 7f3d227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:35.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.547+0000 7f3d227fc640 10 monclient(hunting): my global_id is 4329 2024-03-25T19:37:35.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.547+0000 7f3d227fc640 1 -- 172.21.15.44:0/2367922705 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d10003520 con 0x7f3d240a98f0 2024-03-25T19:37:35.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.612+0000 7f3d227fc640 1 -- 172.21.15.44:0/2367922705 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1735815635 0 0) 0x7f3d240b2a30 con 0x7f3d241430c0 2024-03-25T19:37:35.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.612+0000 7f3d227fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:35.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.612+0000 7f3d227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:35.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.612+0000 7f3d227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:35.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.612+0000 7f3d227fc640 10 monclient(hunting): my global_id is 4301 2024-03-25T19:37:35.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.612+0000 7f3d227fc640 1 -- 172.21.15.44:0/2367922705 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d24142520 con 0x7f3d241430c0 2024-03-25T19:37:35.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.629+0000 7f3d227fc640 1 -- 172.21.15.44:0/2367922705 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2258165688 0 0) 0x7f3d10003520 con 0x7f3d240a98f0 2024-03-25T19:37:35.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.629+0000 7f3d227fc640 1 -- 172.21.15.44:0/2367922705 >> v1:172.21.15.44:6790/0 conn(0x7f3d241430c0 legacy=0x7f3d2414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:35.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.629+0000 7f3d227fc640 1 -- 172.21.15.44:0/2367922705 >> v1:172.21.15.111:6789/0 conn(0x7f3d2413f710 legacy=0x7f3d241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:35.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.629+0000 7f3d227fc640 10 monclient: _finish_hunting 0 2024-03-25T19:37:35.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.629+0000 7f3d227fc640 1 monclient: found mon.noname-a 2024-03-25T19:37:35.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.629+0000 7f3d227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:35.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.629+0000 7f3d227fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:37:35.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.629+0000 7f3d227fc640 1 -- 172.21.15.44:0/2367922705 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d240b44c0 con 0x7f3d240a98f0 2024-03-25T19:37:35.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.629+0000 7f3d227fc640 10 monclient: _finish_auth 0 2024-03-25T19:37:35.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.629+0000 7f3d227fc640 10 monclient: _check_auth_tickets 2024-03-25T19:37:35.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.629+0000 7f3d227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:35.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.629+0000 7f3d227fc640 1 -- 172.21.15.44:0/2367922705 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3d14001690 con 0x7f3d240a98f0 2024-03-25T19:37:35.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.629+0000 7f3d227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:35.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.629+0000 7f3d227fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:35.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.629+0000 7f3d227fc640 10 monclient: dump: 2024-03-25T19:37:35.630 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:35.630 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:35.630 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:35.630 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:35.630 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:35.630 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:35.630 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:35.630 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:35.630 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:35.630 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:35.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.629+0000 7f3d2973d640 5 monclient: authenticate success, global_id 4329 2024-03-25T19:37:35.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.629+0000 7f3d2973d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:35.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.651+0000 7f3d227fc640 1 -- 172.21.15.44:0/2367922705 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d140017f0 con 0x7f3d240a98f0 2024-03-25T19:37:35.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.651+0000 7f3d227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:35.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.651+0000 7f3d227fc640 1 -- 172.21.15.44:0/2367922705 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3d14001c00 con 0x7f3d240a98f0 2024-03-25T19:37:35.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.651+0000 7f3d2973d640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:35.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.651+0000 7f3d2973d640 4 set_mon_vals no callback set 2024-03-25T19:37:35.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.651+0000 7f3d227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:35.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.651+0000 7f3d227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:35.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.651+0000 7f3d227fc640 10 monclient: dump: 2024-03-25T19:37:35.651 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:35.651 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:35.651 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:35.651 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:35.651 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:35.652 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:35.652 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:35.652 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:35.652 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:35.652 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:35.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.651+0000 7f3d2973d640 10 monclient: shutdown 2024-03-25T19:37:35.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.651+0000 7f3d2973d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:35.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.651+0000 7f3d2973d640 1 -- 172.21.15.44:0/2367922705 >> v1:172.21.15.44:6789/0 conn(0x7f3d240a98f0 legacy=0x7f3d240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:35.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.651+0000 7f3d2973d640 1 -- 172.21.15.44:0/2367922705 shutdown_connections 2024-03-25T19:37:35.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.651+0000 7f3d2973d640 1 -- 172.21.15.44:0/2367922705 >> 172.21.15.44:0/2367922705 conn(0x7f3d240b7ea0 msgr2=0x7f3d240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:35.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.652+0000 7f3d2973d640 1 -- 172.21.15.44:0/2367922705 shutdown_connections 2024-03-25T19:37:35.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.652+0000 7f3d2973d640 1 -- 172.21.15.44:0/2367922705 wait complete. 2024-03-25T19:37:35.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.652+0000 7f3d2973d640 10 monclient: build_initial_monmap 2024-03-25T19:37:35.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.652+0000 7f3d2973d640 10 monclient: monmap: 2024-03-25T19:37:35.655 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:37:35.655 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:35.655 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:37:35.655 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:37:35.655 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:35.655 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:37:35.655 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:37:35.655 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:37:35.655 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:37:35.655 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:35.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.653+0000 7f3d2973d640 1 librados: starting msgr at 2024-03-25T19:37:35.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.653+0000 7f3d2973d640 1 librados: starting objecter 2024-03-25T19:37:35.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.653+0000 7f3d2973d640 1 Processor -- start 2024-03-25T19:37:35.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.653+0000 7f3d2973d640 1 -- start start 2024-03-25T19:37:35.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.653+0000 7f3d2973d640 1 librados: setting wanted keys 2024-03-25T19:37:35.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.653+0000 7f3d2973d640 1 librados: calling monclient init 2024-03-25T19:37:35.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.653+0000 7f3d2973d640 10 monclient: init 2024-03-25T19:37:35.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.653+0000 7f3d2973d640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:35.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.653+0000 7f3d2973d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:37:35.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.654+0000 7f3d2973d640 10 monclient(hunting): picked mon.noname-b con 0x7f3d240a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:37:35.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.654+0000 7f3d2973d640 10 monclient(hunting): picked mon.noname-c con 0x7f3d2413f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:37:35.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.654+0000 7f3d2973d640 10 monclient(hunting): picked mon.noname-a con 0x7f3d241430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:37:35.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.654+0000 7f3d2973d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d241db050 con 0x7f3d241430c0 2024-03-25T19:37:35.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.654+0000 7f3d2973d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d241dab70 con 0x7f3d240a98f0 2024-03-25T19:37:35.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.654+0000 7f3d2973d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d241dad00 con 0x7f3d2413f710 2024-03-25T19:37:35.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.654+0000 7f3d2973d640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:35.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.654+0000 7f3d2973d640 10 monclient(hunting): authenticate will time out at 5519.825143s 2024-03-25T19:37:35.657 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.656+0000 7f3d22ffd640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3d240a98f0 0x7f3d241db4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:43420/0 (socket says 172.21.15.44:43420) 2024-03-25T19:37:35.657 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.656+0000 7f3d22ffd640 1 -- 172.21.15.44:0/959529369 learned_addr learned my addr 172.21.15.44:0/959529369 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:37:35.657 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.657+0000 7f3d1b7fe640 10 client.?.objecter ms_handle_connect 0x7f3d240a98f0 2024-03-25T19:37:35.657 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.657+0000 7f3d1b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:35.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.663+0000 7f3d1b7fe640 10 client.?.objecter ms_handle_connect 0x7f3d241430c0 2024-03-25T19:37:35.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.663+0000 7f3d1b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:35.664 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.664+0000 7f3d1b7fe640 1 -- 172.21.15.44:0/959529369 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4032347263 0 0) 0x7f3d241dab70 con 0x7f3d240a98f0 2024-03-25T19:37:35.664 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.664+0000 7f3d1b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:35.664 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.664+0000 7f3d1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:35.664 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.664+0000 7f3d1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:35.665 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.664+0000 7f3d1b7fe640 10 monclient(hunting): my global_id is 4321 2024-03-25T19:37:35.665 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.664+0000 7f3d1b7fe640 1 -- 172.21.15.44:0/959529369 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d04004eb0 con 0x7f3d240a98f0 2024-03-25T19:37:35.665 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.665+0000 7f3d1b7fe640 10 client.?.objecter ms_handle_connect 0x7f3d2413f710 2024-03-25T19:37:35.665 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.665+0000 7f3d1b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:35.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.669+0000 7f3d1b7fe640 1 -- 172.21.15.44:0/959529369 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824251993 0 0) 0x7f3d241db050 con 0x7f3d241430c0 2024-03-25T19:37:35.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.669+0000 7f3d1b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:35.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.669+0000 7f3d1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:35.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.669+0000 7f3d1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:35.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.669+0000 7f3d1b7fe640 10 monclient(hunting): my global_id is 4332 2024-03-25T19:37:35.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.670+0000 7f3d1b7fe640 1 -- 172.21.15.44:0/959529369 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d241dab70 con 0x7f3d241430c0 2024-03-25T19:37:35.683 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.683+0000 7f3d1b7fe640 1 -- 172.21.15.44:0/959529369 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3102817136 0 0) 0x7f3d04004eb0 con 0x7f3d240a98f0 2024-03-25T19:37:35.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.684+0000 7f3d1b7fe640 1 -- 172.21.15.44:0/959529369 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d241db050 con 0x7f3d240a98f0 2024-03-25T19:37:35.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.684+0000 7f3d1b7fe640 1 -- 172.21.15.44:0/959529369 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3d14001690 con 0x7f3d240a98f0 2024-03-25T19:37:35.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.684+0000 7f3d1b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:35.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.684+0000 7f3d1b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:35.684 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.684+0000 7f3d1b7fe640 10 monclient(hunting): dump: 2024-03-25T19:37:35.684 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:35.684 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:35.684 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:35.684 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:35.684 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:35.684 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:35.684 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:35.684 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:35.685 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:35.685 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:35.685 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.684+0000 7f3d1b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:35.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.698+0000 7f3d1b7fe640 1 -- 172.21.15.44:0/959529369 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1020022350 0 0) 0x7f3d241dad00 con 0x7f3d2413f710 2024-03-25T19:37:35.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.698+0000 7f3d1b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:35.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.698+0000 7f3d1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:35.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.698+0000 7f3d1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:35.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.698+0000 7f3d1b7fe640 10 monclient(hunting): my global_id is 4304 2024-03-25T19:37:35.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.698+0000 7f3d1b7fe640 1 -- 172.21.15.44:0/959529369 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d04004eb0 con 0x7f3d2413f710 2024-03-25T19:37:35.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.705+0000 7f3d1b7fe640 1 -- 172.21.15.44:0/959529369 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1598895736 0 0) 0x7f3d241db050 con 0x7f3d240a98f0 2024-03-25T19:37:35.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.705+0000 7f3d1b7fe640 1 -- 172.21.15.44:0/959529369 >> v1:172.21.15.44:6790/0 conn(0x7f3d2413f710 legacy=0x7f3d241d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:35.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.705+0000 7f3d1b7fe640 1 -- 172.21.15.44:0/959529369 >> v1:172.21.15.44:6789/0 conn(0x7f3d241430c0 legacy=0x7f3d241393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:35.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.705+0000 7f3d1b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:37:35.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.705+0000 7f3d1b7fe640 1 monclient: found mon.b 2024-03-25T19:37:35.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.705+0000 7f3d1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:35.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.705+0000 7f3d1b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:35.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.705+0000 7f3d1b7fe640 1 -- 172.21.15.44:0/959529369 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d240b70a0 con 0x7f3d240a98f0 2024-03-25T19:37:35.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.705+0000 7f3d1b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:37:35.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.705+0000 7f3d1b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:37:35.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.705+0000 7f3d1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:35.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.705+0000 7f3d2973d640 5 monclient: authenticate success, global_id 4321 2024-03-25T19:37:35.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.705+0000 7f3d2973d640 10 monclient: _renew_subs 2024-03-25T19:37:35.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.705+0000 7f3d2973d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:35.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.705+0000 7f3d2973d640 1 -- 172.21.15.44:0/959529369 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d240b72f0 con 0x7f3d240a98f0 2024-03-25T19:37:35.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.705+0000 7f3d2973d640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:35.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.705+0000 7f3d2973d640 10 monclient: _renew_subs 2024-03-25T19:37:35.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.705+0000 7f3d2973d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:35.706 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.705+0000 7f3d2973d640 1 -- 172.21.15.44:0/959529369 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d241e32a0 con 0x7f3d240a98f0 2024-03-25T19:37:35.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.705+0000 7f3d2973d640 1 librados: init done 2024-03-25T19:37:35.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.706+0000 7f3d2973d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:35.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.706+0000 7f3d2973d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:35.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.706+0000 7f3d2973d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:35.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.706+0000 7f3d2973d640 1 -- 172.21.15.44:0/959529369 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d240627b0 con 0x7f3d240a98f0 2024-03-25T19:37:35.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.974+0000 7f3d1b7fe640 1 -- 172.21.15.44:0/959529369 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d140038a0 con 0x7f3d240a98f0 2024-03-25T19:37:35.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.974+0000 7f3d1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:35.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.975+0000 7f3d1b7fe640 1 -- 172.21.15.44:0/959529369 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3d14001cc0 con 0x7f3d240a98f0 2024-03-25T19:37:35.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.975+0000 7f3d1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:35.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.975+0000 7f3d1b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:35.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.975+0000 7f3d1b7fe640 10 monclient: dump: 2024-03-25T19:37:35.975 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:35.975 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:35.975 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:35.975 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:35.975 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:35.975 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:35.976 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:35.976 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:35.976 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:35.976 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:35.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.975+0000 7f3d21ffb640 4 set_mon_vals no callback set 2024-03-25T19:37:35.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.975+0000 7f3d1b7fe640 1 -- 172.21.15.44:0/959529369 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f3d14003db0 con 0x7f3d240a98f0 2024-03-25T19:37:35.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.976+0000 7f3d1bfff640 10 monclient: get_auth_request con 0x7f3d0406de00 auth_method 0 2024-03-25T19:37:35.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.976+0000 7f3d1b7fe640 10 client.4321.objecter ms_handle_connect 0x7f3d0406de00 2024-03-25T19:37:35.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.980+0000 7f3d1b7fe640 1 -- 172.21.15.44:0/959529369 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f3d14054910 con 0x7f3d240a98f0 2024-03-25T19:37:35.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.980+0000 7f3d1b7fe640 10 client.4321.objecter ms_dispatch 0x7f3d24000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:35.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.980+0000 7f3d1b7fe640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:35.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.980+0000 7f3d1b7fe640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:35.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.980+0000 7f3d1b7fe640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-25T19:37:35.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.980+0000 7f3d1b7fe640 1 -- 172.21.15.44:0/959529369 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3d14050420 con 0x7f3d240a98f0 2024-03-25T19:37:35.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.980+0000 7f3d1b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:35.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:35.980+0000 7f3d1b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:36.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:36.244+0000 7f3d2973d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:36.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:36.244+0000 7f3d2973d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:36.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:36.244+0000 7f3d2973d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:36.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:36.244+0000 7f3d2973d640 1 -- 172.21.15.44:0/959529369 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d24069f40 con 0x7f3d240a98f0 2024-03-25T19:37:36.272 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:36.272+0000 7f3d1b7fe640 1 -- 172.21.15.44:0/959529369 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f3d14052350 con 0x7f3d240a98f0 2024-03-25T19:37:36.272 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:36.272+0000 7f3d1b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:36.272 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:36.272+0000 7f3d1b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:36.272 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:37:36.272 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:36.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:36.274+0000 7f3d2973d640 10 librados: watch_flush enter 2024-03-25T19:37:36.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:36.274+0000 7f3d2973d640 10 librados: watch_flush exit 2024-03-25T19:37:36.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:36.274+0000 7f3d2973d640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-25T19:37:36.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:36.274+0000 7f3d2973d640 10 client.4321.objecter successfully canceled tick 2024-03-25T19:37:36.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:36.275+0000 7f3d2973d640 1 -- 172.21.15.44:0/959529369 >> v1:172.21.15.111:6800/3883215793 conn(0x7f3d0406de00 legacy=0x7f3d040702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:36.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:36.275+0000 7f3d2973d640 10 monclient: shutdown 2024-03-25T19:37:36.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:36.275+0000 7f3d2973d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:36.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:36.275+0000 7f3d2973d640 1 -- 172.21.15.44:0/959529369 >> v1:172.21.15.111:6789/0 conn(0x7f3d240a98f0 legacy=0x7f3d241db4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:36.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:36.275+0000 7f3d2973d640 1 -- 172.21.15.44:0/959529369 shutdown_connections 2024-03-25T19:37:36.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:36.275+0000 7f3d2973d640 1 -- 172.21.15.44:0/959529369 >> 172.21.15.44:0/959529369 conn(0x7f3d240b7ea0 msgr2=0x7f3d24074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:36.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:36.275+0000 7f3d2973d640 1 -- 172.21.15.44:0/959529369 shutdown_connections 2024-03-25T19:37:36.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:36.275+0000 7f3d2973d640 1 -- 172.21.15.44:0/959529369 wait complete. 2024-03-25T19:37:36.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:36.275+0000 7f3d2973d640 1 librados: shutdown 2024-03-25T19:37:36.288 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:43.094 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:43.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.212+0000 7f8c2ef3d640 10 monclient: get_monmap_and_config 2024-03-25T19:37:43.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.212+0000 7f8c2ef3d640 10 monclient: build_initial_monmap 2024-03-25T19:37:43.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.212+0000 7f8c2ef3d640 10 monclient: monmap: 2024-03-25T19:37:43.212 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:37:43.212 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:43.212 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:37:43.212935+0000 2024-03-25T19:37:43.212 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:37:43.212935+0000 2024-03-25T19:37:43.212 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:43.212 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:43.212 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:37:43.212 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:37:43.212 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:37:43.212 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:43.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.213+0000 7f8c2ef3d640 1 Processor -- start 2024-03-25T19:37:43.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.213+0000 7f8c2ef3d640 1 -- start start 2024-03-25T19:37:43.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.213+0000 7f8c2ef3d640 10 monclient: init 2024-03-25T19:37:43.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.214+0000 7f8c2ef3d640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:43.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.214+0000 7f8c2ef3d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:37:43.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.214+0000 7f8c2ef3d640 10 monclient(hunting): picked mon.noname-c con 0x7f8c2813fc60 addr v1:172.21.15.111:6789/0 2024-03-25T19:37:43.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.214+0000 7f8c2ef3d640 10 monclient(hunting): picked mon.noname-a con 0x7f8c28070570 addr v1:172.21.15.44:6789/0 2024-03-25T19:37:43.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.214+0000 7f8c2ef3d640 10 monclient(hunting): picked mon.noname-b con 0x7f8c280727a0 addr v1:172.21.15.44:6790/0 2024-03-25T19:37:43.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.214+0000 7f8c2ef3d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c2813c9f0 con 0x7f8c28070570 2024-03-25T19:37:43.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.214+0000 7f8c2ef3d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c2806fac0 con 0x7f8c2813fc60 2024-03-25T19:37:43.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.214+0000 7f8c2ef3d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c280715d0 con 0x7f8c280727a0 2024-03-25T19:37:43.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.214+0000 7f8c2ef3d640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:43.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.214+0000 7f8c2ef3d640 10 monclient(hunting): authenticate will time out at 5527.384324s 2024-03-25T19:37:43.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.225+0000 7f8c2ccd9640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f8c2813fc60 0x7f8c2806f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:42820/0 (socket says 172.21.15.44:42820) 2024-03-25T19:37:43.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.225+0000 7f8c2ccd9640 1 -- 172.21.15.44:0/2297626651 learned_addr learned my addr 172.21.15.44:0/2297626651 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:37:43.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.239+0000 7f8c277fe640 1 -- 172.21.15.44:0/2297626651 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1827436866 0 0) 0x7f8c2806fac0 con 0x7f8c2813fc60 2024-03-25T19:37:43.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.239+0000 7f8c277fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:43.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.239+0000 7f8c277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:43.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.239+0000 7f8c277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:43.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.239+0000 7f8c277fe640 10 monclient(hunting): my global_id is 4324 2024-03-25T19:37:43.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.239+0000 7f8c277fe640 1 -- 172.21.15.44:0/2297626651 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c18003520 con 0x7f8c2813fc60 2024-03-25T19:37:43.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.245+0000 7f8c277fe640 1 -- 172.21.15.44:0/2297626651 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 981508571 0 0) 0x7f8c280715d0 con 0x7f8c280727a0 2024-03-25T19:37:43.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.245+0000 7f8c277fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:43.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.245+0000 7f8c277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:43.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.245+0000 7f8c277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:43.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.245+0000 7f8c277fe640 10 monclient(hunting): my global_id is 4307 2024-03-25T19:37:43.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.245+0000 7f8c277fe640 1 -- 172.21.15.44:0/2297626651 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c2806fac0 con 0x7f8c280727a0 2024-03-25T19:37:43.254 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.255+0000 7f8c277fe640 1 -- 172.21.15.44:0/2297626651 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3317625048 0 0) 0x7f8c2813c9f0 con 0x7f8c28070570 2024-03-25T19:37:43.254 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.255+0000 7f8c277fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:43.254 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.255+0000 7f8c277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:43.254 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.255+0000 7f8c277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:43.254 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.255+0000 7f8c277fe640 10 monclient(hunting): my global_id is 4335 2024-03-25T19:37:43.254 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.255+0000 7f8c277fe640 1 -- 172.21.15.44:0/2297626651 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c280715d0 con 0x7f8c28070570 2024-03-25T19:37:43.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.257+0000 7f8c277fe640 1 -- 172.21.15.44:0/2297626651 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1602701523 0 0) 0x7f8c2806fac0 con 0x7f8c280727a0 2024-03-25T19:37:43.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.257+0000 7f8c277fe640 1 -- 172.21.15.44:0/2297626651 >> v1:172.21.15.111:6789/0 conn(0x7f8c2813fc60 legacy=0x7f8c2806f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:43.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.257+0000 7f8c277fe640 1 -- 172.21.15.44:0/2297626651 >> v1:172.21.15.44:6789/0 conn(0x7f8c28070570 legacy=0x7f8c28072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:43.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.257+0000 7f8c277fe640 10 monclient: _finish_hunting 0 2024-03-25T19:37:43.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.257+0000 7f8c277fe640 1 monclient: found mon.noname-b 2024-03-25T19:37:43.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.257+0000 7f8c277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:43.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.257+0000 7f8c277fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:37:43.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.257+0000 7f8c277fe640 1 -- 172.21.15.44:0/2297626651 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c280718b0 con 0x7f8c280727a0 2024-03-25T19:37:43.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.257+0000 7f8c277fe640 10 monclient: _finish_auth 0 2024-03-25T19:37:43.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.257+0000 7f8c277fe640 10 monclient: _check_auth_tickets 2024-03-25T19:37:43.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.257+0000 7f8c277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:43.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.257+0000 7f8c277fe640 1 -- 172.21.15.44:0/2297626651 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f8c200019f0 con 0x7f8c280727a0 2024-03-25T19:37:43.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.257+0000 7f8c2ef3d640 5 monclient: authenticate success, global_id 4307 2024-03-25T19:37:43.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.257+0000 7f8c2ef3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:43.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.257+0000 7f8c277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:43.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.257+0000 7f8c277fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:43.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.257+0000 7f8c277fe640 10 monclient: dump: 2024-03-25T19:37:43.257 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:43.257 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:43.257 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:43.257 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:43.258 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:43.258 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:43.258 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:43.258 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:43.258 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:43.258 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:43.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.257+0000 7f8c2ef3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:43.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.267+0000 7f8c277fe640 1 -- 172.21.15.44:0/2297626651 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c20001c10 con 0x7f8c280727a0 2024-03-25T19:37:43.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.267+0000 7f8c277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:43.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.267+0000 7f8c277fe640 1 -- 172.21.15.44:0/2297626651 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f8c20001fa0 con 0x7f8c280727a0 2024-03-25T19:37:43.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.267+0000 7f8c2ef3d640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:43.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.267+0000 7f8c2ef3d640 4 set_mon_vals no callback set 2024-03-25T19:37:43.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.267+0000 7f8c277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:43.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.267+0000 7f8c277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:43.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.267+0000 7f8c277fe640 10 monclient: dump: 2024-03-25T19:37:43.266 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:43.266 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:43.266 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:43.266 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:43.267 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:43.267 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:43.267 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:43.267 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:43.267 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:43.267 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:43.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.267+0000 7f8c2ef3d640 10 monclient: shutdown 2024-03-25T19:37:43.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.267+0000 7f8c2ef3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:43.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.267+0000 7f8c2ef3d640 1 -- 172.21.15.44:0/2297626651 >> v1:172.21.15.44:6790/0 conn(0x7f8c280727a0 legacy=0x7f8c2814a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:43.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.268+0000 7f8c2ef3d640 1 -- 172.21.15.44:0/2297626651 shutdown_connections 2024-03-25T19:37:43.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.268+0000 7f8c2ef3d640 1 -- 172.21.15.44:0/2297626651 >> 172.21.15.44:0/2297626651 conn(0x7f8c280b4a80 msgr2=0x7f8c280b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:43.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.268+0000 7f8c2ef3d640 1 -- 172.21.15.44:0/2297626651 shutdown_connections 2024-03-25T19:37:43.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.268+0000 7f8c2ef3d640 1 -- 172.21.15.44:0/2297626651 wait complete. 2024-03-25T19:37:43.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.268+0000 7f8c2ef3d640 10 monclient: build_initial_monmap 2024-03-25T19:37:43.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.268+0000 7f8c2ef3d640 10 monclient: monmap: 2024-03-25T19:37:43.270 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:37:43.270 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:43.270 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:37:43.270 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:37:43.270 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:43.270 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:37:43.270 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:37:43.270 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:37:43.270 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:37:43.270 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:43.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.269+0000 7f8c2ef3d640 1 librados: starting msgr at 2024-03-25T19:37:43.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.269+0000 7f8c2ef3d640 1 librados: starting objecter 2024-03-25T19:37:43.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.269+0000 7f8c2ef3d640 1 Processor -- start 2024-03-25T19:37:43.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.269+0000 7f8c2ef3d640 1 -- start start 2024-03-25T19:37:43.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.269+0000 7f8c2ef3d640 1 librados: setting wanted keys 2024-03-25T19:37:43.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.269+0000 7f8c2ef3d640 1 librados: calling monclient init 2024-03-25T19:37:43.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.269+0000 7f8c2ef3d640 10 monclient: init 2024-03-25T19:37:43.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.269+0000 7f8c2ef3d640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:43.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.269+0000 7f8c2ef3d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:37:43.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.269+0000 7f8c2ef3d640 10 monclient(hunting): picked mon.noname-a con 0x7f8c28070570 addr v1:172.21.15.44:6789/0 2024-03-25T19:37:43.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.270+0000 7f8c2ef3d640 10 monclient(hunting): picked mon.noname-c con 0x7f8c280727a0 addr v1:172.21.15.44:6790/0 2024-03-25T19:37:43.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.270+0000 7f8c2ef3d640 10 monclient(hunting): picked mon.noname-b con 0x7f8c2813fc60 addr v1:172.21.15.111:6789/0 2024-03-25T19:37:43.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.270+0000 7f8c2ef3d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c28142cf0 con 0x7f8c28070570 2024-03-25T19:37:43.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.270+0000 7f8c2ef3d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c281429c0 con 0x7f8c2813fc60 2024-03-25T19:37:43.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.270+0000 7f8c2ef3d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c281db1e0 con 0x7f8c280727a0 2024-03-25T19:37:43.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.270+0000 7f8c2ef3d640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:43.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.270+0000 7f8c2ef3d640 10 monclient(hunting): authenticate will time out at 5527.439893s 2024-03-25T19:37:43.286 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.287+0000 7f8c2ccd9640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f8c28070570 0x7f8c281daad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54498/0 (socket says 172.21.15.44:54498) 2024-03-25T19:37:43.286 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.287+0000 7f8c2ccd9640 1 -- 172.21.15.44:0/2865335154 learned_addr learned my addr 172.21.15.44:0/2865335154 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:37:43.287 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.288+0000 7f8c257fa640 10 client.?.objecter ms_handle_connect 0x7f8c28070570 2024-03-25T19:37:43.287 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.288+0000 7f8c257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:43.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.297+0000 7f8c257fa640 10 client.?.objecter ms_handle_connect 0x7f8c2813fc60 2024-03-25T19:37:43.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.297+0000 7f8c257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:43.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.299+0000 7f8c257fa640 1 -- 172.21.15.44:0/2865335154 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375266266 0 0) 0x7f8c28142cf0 con 0x7f8c28070570 2024-03-25T19:37:43.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.299+0000 7f8c257fa640 10 monclient(hunting): _negotiate 2024-03-25T19:37:43.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.299+0000 7f8c257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:43.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.299+0000 7f8c257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:43.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.299+0000 7f8c257fa640 10 monclient(hunting): my global_id is 4338 2024-03-25T19:37:43.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.299+0000 7f8c257fa640 1 -- 172.21.15.44:0/2865335154 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c0c004eb0 con 0x7f8c28070570 2024-03-25T19:37:43.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.302+0000 7f8c257fa640 10 client.?.objecter ms_handle_connect 0x7f8c280727a0 2024-03-25T19:37:43.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.302+0000 7f8c257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:43.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.305+0000 7f8c257fa640 1 -- 172.21.15.44:0/2865335154 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908978344 0 0) 0x7f8c281429c0 con 0x7f8c2813fc60 2024-03-25T19:37:43.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.305+0000 7f8c257fa640 10 monclient(hunting): _negotiate 2024-03-25T19:37:43.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.305+0000 7f8c257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:43.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.305+0000 7f8c257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:43.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.305+0000 7f8c257fa640 10 monclient(hunting): my global_id is 4327 2024-03-25T19:37:43.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.305+0000 7f8c257fa640 1 -- 172.21.15.44:0/2865335154 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c28142cf0 con 0x7f8c2813fc60 2024-03-25T19:37:43.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.311+0000 7f8c257fa640 1 -- 172.21.15.44:0/2865335154 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2090482330 0 0) 0x7f8c0c004eb0 con 0x7f8c28070570 2024-03-25T19:37:43.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.311+0000 7f8c257fa640 1 -- 172.21.15.44:0/2865335154 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c281429c0 con 0x7f8c28070570 2024-03-25T19:37:43.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.311+0000 7f8c257fa640 1 -- 172.21.15.44:0/2865335154 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f8c1c001830 con 0x7f8c28070570 2024-03-25T19:37:43.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.311+0000 7f8c257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:43.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.311+0000 7f8c257fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:43.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.311+0000 7f8c257fa640 10 monclient(hunting): dump: 2024-03-25T19:37:43.311 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:43.311 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:43.311 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:43.311 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:43.311 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:43.311 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:43.311 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:43.311 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:43.311 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:43.312 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:43.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.311+0000 7f8c257fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:43.315 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.316+0000 7f8c257fa640 1 -- 172.21.15.44:0/2865335154 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2846368006 0 0) 0x7f8c281db1e0 con 0x7f8c280727a0 2024-03-25T19:37:43.315 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.316+0000 7f8c257fa640 10 monclient(hunting): _negotiate 2024-03-25T19:37:43.315 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.316+0000 7f8c257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:43.315 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.316+0000 7f8c257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:43.315 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.316+0000 7f8c257fa640 10 monclient(hunting): my global_id is 4310 2024-03-25T19:37:43.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.316+0000 7f8c257fa640 1 -- 172.21.15.44:0/2865335154 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c0c004eb0 con 0x7f8c280727a0 2024-03-25T19:37:43.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.322+0000 7f8c257fa640 1 -- 172.21.15.44:0/2865335154 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3987409621 0 0) 0x7f8c281429c0 con 0x7f8c28070570 2024-03-25T19:37:43.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.322+0000 7f8c257fa640 1 -- 172.21.15.44:0/2865335154 >> v1:172.21.15.44:6790/0 conn(0x7f8c280727a0 legacy=0x7f8c281340e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:43.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.322+0000 7f8c257fa640 1 -- 172.21.15.44:0/2865335154 >> v1:172.21.15.111:6789/0 conn(0x7f8c2813fc60 legacy=0x7f8c281e2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:43.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.322+0000 7f8c257fa640 10 monclient: _finish_hunting 0 2024-03-25T19:37:43.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.322+0000 7f8c257fa640 1 monclient: found mon.a 2024-03-25T19:37:43.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.322+0000 7f8c257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:43.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.322+0000 7f8c257fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:37:43.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.322+0000 7f8c257fa640 1 -- 172.21.15.44:0/2865335154 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c281db370 con 0x7f8c28070570 2024-03-25T19:37:43.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.322+0000 7f8c257fa640 10 monclient: _finish_auth 0 2024-03-25T19:37:43.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.322+0000 7f8c257fa640 10 monclient: _check_auth_tickets 2024-03-25T19:37:43.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.322+0000 7f8c257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:43.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.322+0000 7f8c2ef3d640 5 monclient: authenticate success, global_id 4338 2024-03-25T19:37:43.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.322+0000 7f8c2ef3d640 10 monclient: _renew_subs 2024-03-25T19:37:43.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.322+0000 7f8c2ef3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:37:43.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.322+0000 7f8c2ef3d640 1 -- 172.21.15.44:0/2865335154 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c28143180 con 0x7f8c28070570 2024-03-25T19:37:43.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.322+0000 7f8c2ef3d640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:43.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.322+0000 7f8c2ef3d640 10 monclient: _renew_subs 2024-03-25T19:37:43.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.322+0000 7f8c2ef3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:37:43.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.322+0000 7f8c2ef3d640 1 -- 172.21.15.44:0/2865335154 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c2813a810 con 0x7f8c28070570 2024-03-25T19:37:43.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.322+0000 7f8c2ef3d640 1 librados: init done 2024-03-25T19:37:43.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.323+0000 7f8c2ef3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:43.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.323+0000 7f8c2ef3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:43.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.323+0000 7f8c2ef3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:37:43.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.324+0000 7f8c2ef3d640 1 -- 172.21.15.44:0/2865335154 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c280b30e0 con 0x7f8c28070570 2024-03-25T19:37:43.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.641+0000 7f8c257fa640 1 -- 172.21.15.44:0/2865335154 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c1c001ce0 con 0x7f8c28070570 2024-03-25T19:37:43.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.641+0000 7f8c257fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:43.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.641+0000 7f8c257fa640 1 -- 172.21.15.44:0/2865335154 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f8c1c004980 con 0x7f8c28070570 2024-03-25T19:37:43.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.641+0000 7f8c257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:43.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.641+0000 7f8c257fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:43.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.641+0000 7f8c257fa640 10 monclient: dump: 2024-03-25T19:37:43.641 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:43.641 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:43.641 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:43.641 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:43.641 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:43.641 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:43.641 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:43.641 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:43.641 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:43.641 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:43.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.641+0000 7f8c26ffd640 4 set_mon_vals no callback set 2024-03-25T19:37:43.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.642+0000 7f8c257fa640 1 -- 172.21.15.44:0/2865335154 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f8c1c004ba0 con 0x7f8c28070570 2024-03-25T19:37:43.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.643+0000 7f8c27fff640 10 monclient: get_auth_request con 0x7f8c0c06ddb0 auth_method 0 2024-03-25T19:37:43.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.643+0000 7f8c257fa640 10 client.4338.objecter ms_handle_connect 0x7f8c0c06ddb0 2024-03-25T19:37:43.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.649+0000 7f8c257fa640 1 -- 172.21.15.44:0/2865335154 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f8c1c0547e0 con 0x7f8c28070570 2024-03-25T19:37:43.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.649+0000 7f8c257fa640 10 client.4338.objecter ms_dispatch 0x7f8c28000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:43.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.649+0000 7f8c257fa640 3 client.4338.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:43.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.649+0000 7f8c257fa640 3 client.4338.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:43.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.649+0000 7f8c257fa640 20 client.4338.objecter dump_active .. 0 homeless 2024-03-25T19:37:43.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.649+0000 7f8c257fa640 1 -- 172.21.15.44:0/2865335154 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8c1c050120 con 0x7f8c28070570 2024-03-25T19:37:43.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.649+0000 7f8c257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:43.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.649+0000 7f8c257fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:43.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.872+0000 7f8c2ef3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:43.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.872+0000 7f8c2ef3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:43.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.872+0000 7f8c2ef3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:37:43.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.872+0000 7f8c2ef3d640 1 -- 172.21.15.44:0/2865335154 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c280627b0 con 0x7f8c28070570 2024-03-25T19:37:43.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.894+0000 7f8c257fa640 1 -- 172.21.15.44:0/2865335154 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f8c1c052220 con 0x7f8c28070570 2024-03-25T19:37:43.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.894+0000 7f8c257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:43.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.894+0000 7f8c257fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:43.893 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:37:43.893 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:43.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.896+0000 7f8c2ef3d640 10 librados: watch_flush enter 2024-03-25T19:37:43.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.896+0000 7f8c2ef3d640 10 librados: watch_flush exit 2024-03-25T19:37:43.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.896+0000 7f8c2ef3d640 20 client.4338.objecter shutdown clearing up homeless session... 2024-03-25T19:37:43.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.896+0000 7f8c2ef3d640 10 client.4338.objecter successfully canceled tick 2024-03-25T19:37:43.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.896+0000 7f8c2ef3d640 1 -- 172.21.15.44:0/2865335154 >> v1:172.21.15.111:6800/3883215793 conn(0x7f8c0c06ddb0 legacy=0x7f8c0c070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:43.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.896+0000 7f8c2ef3d640 10 monclient: shutdown 2024-03-25T19:37:43.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.896+0000 7f8c2ef3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:43.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.896+0000 7f8c2ef3d640 1 -- 172.21.15.44:0/2865335154 >> v1:172.21.15.44:6789/0 conn(0x7f8c28070570 legacy=0x7f8c281daad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:43.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.896+0000 7f8c2ef3d640 1 -- 172.21.15.44:0/2865335154 shutdown_connections 2024-03-25T19:37:43.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.896+0000 7f8c2ef3d640 1 -- 172.21.15.44:0/2865335154 >> 172.21.15.44:0/2865335154 conn(0x7f8c280b4a80 msgr2=0x7f8c280afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:43.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.896+0000 7f8c2ef3d640 1 -- 172.21.15.44:0/2865335154 shutdown_connections 2024-03-25T19:37:43.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.896+0000 7f8c2ef3d640 1 -- 172.21.15.44:0/2865335154 wait complete. 2024-03-25T19:37:43.896 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:43.896+0000 7f8c2ef3d640 1 librados: shutdown 2024-03-25T19:37:43.905 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:43.980 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:02:34.882 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:43.993 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:02:34.895 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:44.021 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:02:34.922 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:44.032 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:02:34.934 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:44.038 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:02:34.940 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:44.052 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:02:34.932 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:44.056 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:02:34.936 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:44.064 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:02:34.966 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:44.087 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:02:34.967 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:50.712 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:50.811 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.811+0000 7ff9270f4640 10 monclient: get_monmap_and_config 2024-03-25T19:37:50.811 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.811+0000 7ff9270f4640 10 monclient: build_initial_monmap 2024-03-25T19:37:50.811 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.811+0000 7ff9270f4640 10 monclient: monmap: 2024-03-25T19:37:50.812 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:37:50.812 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:50.812 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:37:50.813005+0000 2024-03-25T19:37:50.812 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:37:50.813005+0000 2024-03-25T19:37:50.812 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:50.812 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:50.812 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:37:50.812 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:37:50.812 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:37:50.812 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:50.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.812+0000 7ff9270f4640 1 Processor -- start 2024-03-25T19:37:50.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.812+0000 7ff9270f4640 1 -- start start 2024-03-25T19:37:50.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.812+0000 7ff9270f4640 10 monclient: init 2024-03-25T19:37:50.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.813+0000 7ff9270f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:50.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.813+0000 7ff9270f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:37:50.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.813+0000 7ff9270f4640 10 monclient(hunting): picked mon.noname-c con 0x7ff9200b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T19:37:50.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.813+0000 7ff9270f4640 10 monclient(hunting): picked mon.noname-b con 0x7ff9200b1730 addr v1:172.21.15.44:6790/0 2024-03-25T19:37:50.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.813+0000 7ff9270f4640 10 monclient(hunting): picked mon.noname-a con 0x7ff92014be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:37:50.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.813+0000 7ff9270f4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9200af940 con 0x7ff92014be90 2024-03-25T19:37:50.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.813+0000 7ff9270f4640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9200b0a70 con 0x7ff9200b4d60 2024-03-25T19:37:50.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.813+0000 7ff9270f4640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9200b0d50 con 0x7ff9200b1730 2024-03-25T19:37:50.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.813+0000 7ff9270f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:50.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.813+0000 7ff9270f4640 10 monclient(hunting): authenticate will time out at 5534.984572s 2024-03-25T19:37:50.817 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.816+0000 7ff924e90640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff9200b4d60 0x7ff92013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:41764/0 (socket says 172.21.15.44:41764) 2024-03-25T19:37:50.817 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.816+0000 7ff924e90640 1 -- 172.21.15.44:0/2116733619 learned_addr learned my addr 172.21.15.44:0/2116733619 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:37:50.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.828+0000 7ff91f7fe640 1 -- 172.21.15.44:0/2116733619 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34565013 0 0) 0x7ff9200b0a70 con 0x7ff9200b4d60 2024-03-25T19:37:50.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.828+0000 7ff91f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:50.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.828+0000 7ff91f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:50.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.828+0000 7ff91f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:50.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.828+0000 7ff91f7fe640 10 monclient(hunting): my global_id is 4330 2024-03-25T19:37:50.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.828+0000 7ff91f7fe640 1 -- 172.21.15.44:0/2116733619 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff910003520 con 0x7ff9200b4d60 2024-03-25T19:37:50.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.844+0000 7ff91f7fe640 1 -- 172.21.15.44:0/2116733619 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797234500 0 0) 0x7ff9200b0d50 con 0x7ff9200b1730 2024-03-25T19:37:50.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.844+0000 7ff91f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:50.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.844+0000 7ff91f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:50.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.844+0000 7ff91f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:50.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.844+0000 7ff91f7fe640 10 monclient(hunting): my global_id is 4313 2024-03-25T19:37:50.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.844+0000 7ff91f7fe640 1 -- 172.21.15.44:0/2116733619 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9200b0a70 con 0x7ff9200b1730 2024-03-25T19:37:50.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.848+0000 7ff91f7fe640 1 -- 172.21.15.44:0/2116733619 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3800242680 0 0) 0x7ff910003520 con 0x7ff9200b4d60 2024-03-25T19:37:50.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.848+0000 7ff91f7fe640 1 -- 172.21.15.44:0/2116733619 >> v1:172.21.15.44:6790/0 conn(0x7ff9200b1730 legacy=0x7ff9200b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:50.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.848+0000 7ff91f7fe640 1 -- 172.21.15.44:0/2116733619 >> v1:172.21.15.44:6789/0 conn(0x7ff92014be90 legacy=0x7ff92014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:50.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.848+0000 7ff91f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:37:50.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.848+0000 7ff91f7fe640 1 monclient: found mon.noname-c 2024-03-25T19:37:50.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.848+0000 7ff91f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:50.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.848+0000 7ff91f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:37:50.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.848+0000 7ff91f7fe640 1 -- 172.21.15.44:0/2116733619 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9200b1030 con 0x7ff9200b4d60 2024-03-25T19:37:50.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.848+0000 7ff91f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:37:50.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.848+0000 7ff91f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:37:50.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.848+0000 7ff91f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:50.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.848+0000 7ff91f7fe640 1 -- 172.21.15.44:0/2116733619 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff914001690 con 0x7ff9200b4d60 2024-03-25T19:37:50.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.848+0000 7ff91f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:50.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.848+0000 7ff91f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:50.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.848+0000 7ff91f7fe640 10 monclient: dump: 2024-03-25T19:37:50.849 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:50.849 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:50.849 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:50.849 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:50.849 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:50.849 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:50.849 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:50.849 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:50.849 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:50.850 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:50.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.848+0000 7ff9270f4640 5 monclient: authenticate success, global_id 4330 2024-03-25T19:37:50.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.848+0000 7ff9270f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:50.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.869+0000 7ff91f7fe640 1 -- 172.21.15.44:0/2116733619 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9140017f0 con 0x7ff9200b4d60 2024-03-25T19:37:50.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.869+0000 7ff91f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:50.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.869+0000 7ff91f7fe640 1 -- 172.21.15.44:0/2116733619 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff914001c00 con 0x7ff9200b4d60 2024-03-25T19:37:50.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.869+0000 7ff91f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:50.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.869+0000 7ff91f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:50.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.869+0000 7ff91f7fe640 10 monclient: dump: 2024-03-25T19:37:50.869 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:50.869 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:50.869 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:50.869 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:50.869 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:50.870 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:50.870 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:50.870 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:50.870 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:50.870 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:50.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.869+0000 7ff9270f4640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:50.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.869+0000 7ff9270f4640 4 set_mon_vals no callback set 2024-03-25T19:37:50.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.870+0000 7ff9270f4640 10 monclient: shutdown 2024-03-25T19:37:50.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.870+0000 7ff9270f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:50.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.870+0000 7ff9270f4640 1 -- 172.21.15.44:0/2116733619 >> v1:172.21.15.111:6789/0 conn(0x7ff9200b4d60 legacy=0x7ff92013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:50.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.870+0000 7ff9270f4640 1 -- 172.21.15.44:0/2116733619 shutdown_connections 2024-03-25T19:37:50.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.870+0000 7ff9270f4640 1 -- 172.21.15.44:0/2116733619 >> 172.21.15.44:0/2116733619 conn(0x7ff9200617a0 msgr2=0x7ff920061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:50.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.870+0000 7ff9270f4640 1 -- 172.21.15.44:0/2116733619 shutdown_connections 2024-03-25T19:37:50.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.870+0000 7ff9270f4640 1 -- 172.21.15.44:0/2116733619 wait complete. 2024-03-25T19:37:50.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.870+0000 7ff9270f4640 10 monclient: build_initial_monmap 2024-03-25T19:37:50.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.870+0000 7ff9270f4640 10 monclient: monmap: 2024-03-25T19:37:50.872 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:37:50.872 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:50.872 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:37:50.872 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:37:50.872 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:50.872 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:37:50.872 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:37:50.872 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:37:50.872 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:37:50.872 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:50.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.871+0000 7ff9270f4640 1 librados: starting msgr at 2024-03-25T19:37:50.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.871+0000 7ff9270f4640 1 librados: starting objecter 2024-03-25T19:37:50.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.871+0000 7ff9270f4640 1 Processor -- start 2024-03-25T19:37:50.873 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.871+0000 7ff9270f4640 1 -- start start 2024-03-25T19:37:50.873 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.871+0000 7ff9270f4640 1 librados: setting wanted keys 2024-03-25T19:37:50.873 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.871+0000 7ff9270f4640 1 librados: calling monclient init 2024-03-25T19:37:50.873 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.871+0000 7ff9270f4640 10 monclient: init 2024-03-25T19:37:50.873 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.872+0000 7ff9270f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:50.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.872+0000 7ff9270f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:37:50.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.872+0000 7ff9270f4640 10 monclient(hunting): picked mon.noname-a con 0x7ff9200b1730 addr v1:172.21.15.44:6789/0 2024-03-25T19:37:50.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.872+0000 7ff9270f4640 10 monclient(hunting): picked mon.noname-c con 0x7ff9200b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T19:37:50.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.872+0000 7ff9270f4640 10 monclient(hunting): picked mon.noname-b con 0x7ff92014be90 addr v1:172.21.15.111:6789/0 2024-03-25T19:37:50.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.872+0000 7ff9270f4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff920141f40 con 0x7ff9200b1730 2024-03-25T19:37:50.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.872+0000 7ff9270f4640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff920142360 con 0x7ff92014be90 2024-03-25T19:37:50.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.872+0000 7ff9270f4640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9201424f0 con 0x7ff9200b4d60 2024-03-25T19:37:50.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.872+0000 7ff9270f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:50.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.872+0000 7ff9270f4640 10 monclient(hunting): authenticate will time out at 5535.043189s 2024-03-25T19:37:50.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.879+0000 7ff91ffff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7ff9200b4d60 0x7ff9201d7c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:35716/0 (socket says 172.21.15.44:35716) 2024-03-25T19:37:50.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.879+0000 7ff91ffff640 1 -- 172.21.15.44:0/4152785495 learned_addr learned my addr 172.21.15.44:0/4152785495 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:37:50.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.880+0000 7ff91d7fa640 10 client.?.objecter ms_handle_connect 0x7ff9200b4d60 2024-03-25T19:37:50.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.880+0000 7ff91d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:50.882 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.882+0000 7ff91d7fa640 10 client.?.objecter ms_handle_connect 0x7ff92014be90 2024-03-25T19:37:50.882 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.882+0000 7ff91d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:50.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.886+0000 7ff91d7fa640 1 -- 172.21.15.44:0/4152785495 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647450410 0 0) 0x7ff9200b0a70 con 0x7ff9200b4d60 2024-03-25T19:37:50.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.886+0000 7ff91d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:37:50.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.886+0000 7ff91d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:50.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.886+0000 7ff91d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:50.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.886+0000 7ff91d7fa640 10 monclient(hunting): my global_id is 4316 2024-03-25T19:37:50.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.886+0000 7ff91d7fa640 1 -- 172.21.15.44:0/4152785495 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff904004eb0 con 0x7ff9200b4d60 2024-03-25T19:37:50.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.891+0000 7ff91d7fa640 10 client.?.objecter ms_handle_connect 0x7ff9200b1730 2024-03-25T19:37:50.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.891+0000 7ff91d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:50.892 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.892+0000 7ff91d7fa640 1 -- 172.21.15.44:0/4152785495 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185902141 0 0) 0x7ff920142360 con 0x7ff92014be90 2024-03-25T19:37:50.892 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.892+0000 7ff91d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:37:50.892 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.892+0000 7ff91d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:50.892 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.892+0000 7ff91d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:50.892 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.892+0000 7ff91d7fa640 10 monclient(hunting): my global_id is 4333 2024-03-25T19:37:50.892 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.892+0000 7ff91d7fa640 1 -- 172.21.15.44:0/4152785495 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9200b0a70 con 0x7ff92014be90 2024-03-25T19:37:50.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.903+0000 7ff91d7fa640 1 -- 172.21.15.44:0/4152785495 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 452123722 0 0) 0x7ff904004eb0 con 0x7ff9200b4d60 2024-03-25T19:37:50.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.903+0000 7ff91d7fa640 1 -- 172.21.15.44:0/4152785495 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff920142360 con 0x7ff9200b4d60 2024-03-25T19:37:50.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.903+0000 7ff91d7fa640 1 -- 172.21.15.44:0/4152785495 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff90c001870 con 0x7ff9200b4d60 2024-03-25T19:37:50.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.903+0000 7ff91d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:50.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.903+0000 7ff91d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:50.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.903+0000 7ff91d7fa640 10 monclient(hunting): dump: 2024-03-25T19:37:50.903 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:50.903 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:50.903 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:50.903 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:50.903 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:50.903 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:50.904 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:50.904 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:50.904 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:50.904 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:50.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.903+0000 7ff91d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:50.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.909+0000 7ff91d7fa640 1 -- 172.21.15.44:0/4152785495 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957655613 0 0) 0x7ff920141f40 con 0x7ff9200b1730 2024-03-25T19:37:50.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.909+0000 7ff91d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:37:50.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.909+0000 7ff91d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:50.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.909+0000 7ff91d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:50.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.909+0000 7ff91d7fa640 10 monclient(hunting): my global_id is 4344 2024-03-25T19:37:50.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.910+0000 7ff91d7fa640 1 -- 172.21.15.44:0/4152785495 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff904004eb0 con 0x7ff9200b1730 2024-03-25T19:37:50.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.910+0000 7ff91d7fa640 1 -- 172.21.15.44:0/4152785495 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2892030697 0 0) 0x7ff9200b0a70 con 0x7ff92014be90 2024-03-25T19:37:50.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.910+0000 7ff91d7fa640 1 -- 172.21.15.44:0/4152785495 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff920141f40 con 0x7ff92014be90 2024-03-25T19:37:50.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.910+0000 7ff91d7fa640 1 -- 172.21.15.44:0/4152785495 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff918001b10 con 0x7ff92014be90 2024-03-25T19:37:50.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.910+0000 7ff91d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:50.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.910+0000 7ff91d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:50.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.910+0000 7ff91d7fa640 10 monclient(hunting): dump: 2024-03-25T19:37:50.911 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:50.911 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:50.911 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:50.911 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:50.911 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:50.911 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:50.911 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:50.912 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:50.912 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:50.912 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:50.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.917+0000 7ff91d7fa640 1 -- 172.21.15.44:0/4152785495 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2014627816 0 0) 0x7ff920141f40 con 0x7ff92014be90 2024-03-25T19:37:50.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.917+0000 7ff91d7fa640 1 -- 172.21.15.44:0/4152785495 >> v1:172.21.15.44:6790/0 conn(0x7ff9200b4d60 legacy=0x7ff9201d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:50.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.917+0000 7ff91d7fa640 1 -- 172.21.15.44:0/4152785495 >> v1:172.21.15.44:6789/0 conn(0x7ff9200b1730 legacy=0x7ff9201405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:50.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.917+0000 7ff91d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:37:50.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.917+0000 7ff91d7fa640 1 monclient: found mon.b 2024-03-25T19:37:50.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.917+0000 7ff91d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:50.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.917+0000 7ff91d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:50.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.917+0000 7ff91d7fa640 1 -- 172.21.15.44:0/4152785495 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff920140d00 con 0x7ff92014be90 2024-03-25T19:37:50.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.917+0000 7ff91d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:37:50.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.917+0000 7ff91d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:37:50.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.917+0000 7ff91d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:50.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.917+0000 7ff9270f4640 5 monclient: authenticate success, global_id 4333 2024-03-25T19:37:50.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.917+0000 7ff9270f4640 10 monclient: _renew_subs 2024-03-25T19:37:50.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.917+0000 7ff9270f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:50.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.917+0000 7ff9270f4640 1 -- 172.21.15.44:0/4152785495 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff920138150 con 0x7ff92014be90 2024-03-25T19:37:50.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.917+0000 7ff9270f4640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:50.919 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.917+0000 7ff9270f4640 10 monclient: _renew_subs 2024-03-25T19:37:50.919 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.917+0000 7ff9270f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:50.919 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.917+0000 7ff9270f4640 1 -- 172.21.15.44:0/4152785495 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9201d8520 con 0x7ff92014be90 2024-03-25T19:37:50.919 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.917+0000 7ff9270f4640 1 librados: init done 2024-03-25T19:37:50.919 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.919+0000 7ff9270f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:50.919 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.919+0000 7ff9270f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:50.919 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.919+0000 7ff9270f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:50.919 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:50.919+0000 7ff9270f4640 1 -- 172.21.15.44:0/4152785495 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff920140f50 con 0x7ff92014be90 2024-03-25T19:37:51.192 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.192+0000 7ff91d7fa640 1 -- 172.21.15.44:0/4152785495 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff918002000 con 0x7ff92014be90 2024-03-25T19:37:51.192 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.192+0000 7ff91d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:51.192 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.192+0000 7ff91d7fa640 1 -- 172.21.15.44:0/4152785495 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff918003b30 con 0x7ff92014be90 2024-03-25T19:37:51.192 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.192+0000 7ff91d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:51.192 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.192+0000 7ff91d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:51.192 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.192+0000 7ff91d7fa640 10 monclient: dump: 2024-03-25T19:37:51.193 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:51.193 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:51.193 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:51.193 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:51.193 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:51.193 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:51.193 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:51.193 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:51.193 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:51.193 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:51.193 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.192+0000 7ff91effd640 4 set_mon_vals no callback set 2024-03-25T19:37:51.194 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.193+0000 7ff91d7fa640 1 -- 172.21.15.44:0/4152785495 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7ff918003d50 con 0x7ff92014be90 2024-03-25T19:37:51.194 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.193+0000 7ff924e90640 10 monclient: get_auth_request con 0x7ff90406df50 auth_method 0 2024-03-25T19:37:51.194 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.194+0000 7ff91d7fa640 10 client.4333.objecter ms_handle_connect 0x7ff90406df50 2024-03-25T19:37:51.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.197+0000 7ff91d7fa640 1 -- 172.21.15.44:0/4152785495 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7ff918053760 con 0x7ff92014be90 2024-03-25T19:37:51.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.197+0000 7ff91d7fa640 10 client.4333.objecter ms_dispatch 0x7ff920000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:51.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.197+0000 7ff91d7fa640 3 client.4333.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:51.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.197+0000 7ff91d7fa640 3 client.4333.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:51.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.197+0000 7ff91d7fa640 20 client.4333.objecter dump_active .. 0 homeless 2024-03-25T19:37:51.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.198+0000 7ff91d7fa640 1 -- 172.21.15.44:0/4152785495 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff91804f270 con 0x7ff92014be90 2024-03-25T19:37:51.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.198+0000 7ff91d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:51.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.198+0000 7ff91d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:51.452 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.452+0000 7ff9270f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:51.452 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.452+0000 7ff9270f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:51.452 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.452+0000 7ff9270f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:51.452 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.452+0000 7ff9270f4640 1 -- 172.21.15.44:0/4152785495 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9201d7950 con 0x7ff92014be90 2024-03-25T19:37:51.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.476+0000 7ff91d7fa640 1 -- 172.21.15.44:0/4152785495 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7ff9201d7950 con 0x7ff92014be90 2024-03-25T19:37:51.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.476+0000 7ff91d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:51.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.476+0000 7ff91d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:51.476 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:37:51.476 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:51.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.478+0000 7ff9270f4640 10 librados: watch_flush enter 2024-03-25T19:37:51.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.478+0000 7ff9270f4640 10 librados: watch_flush exit 2024-03-25T19:37:51.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.478+0000 7ff9270f4640 20 client.4333.objecter shutdown clearing up homeless session... 2024-03-25T19:37:51.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.478+0000 7ff9270f4640 10 client.4333.objecter successfully canceled tick 2024-03-25T19:37:51.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.478+0000 7ff9270f4640 1 -- 172.21.15.44:0/4152785495 >> v1:172.21.15.111:6800/3883215793 conn(0x7ff90406df50 legacy=0x7ff904070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:51.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.478+0000 7ff9270f4640 10 monclient: shutdown 2024-03-25T19:37:51.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.478+0000 7ff9270f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:51.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.478+0000 7ff9270f4640 1 -- 172.21.15.44:0/4152785495 >> v1:172.21.15.111:6789/0 conn(0x7ff92014be90 legacy=0x7ff9201e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:51.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.479+0000 7ff9270f4640 1 -- 172.21.15.44:0/4152785495 shutdown_connections 2024-03-25T19:37:51.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.479+0000 7ff9270f4640 1 -- 172.21.15.44:0/4152785495 >> 172.21.15.44:0/4152785495 conn(0x7ff9200617a0 msgr2=0x7ff9200b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:51.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.479+0000 7ff9270f4640 1 -- 172.21.15.44:0/4152785495 shutdown_connections 2024-03-25T19:37:51.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.479+0000 7ff9270f4640 1 -- 172.21.15.44:0/4152785495 wait complete. 2024-03-25T19:37:51.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:51.479+0000 7ff9270f4640 1 librados: shutdown 2024-03-25T19:37:51.490 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:58.296 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:58.412 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.412+0000 7f821e827640 10 monclient: get_monmap_and_config 2024-03-25T19:37:58.412 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.412+0000 7f821e827640 10 monclient: build_initial_monmap 2024-03-25T19:37:58.412 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.412+0000 7f821e827640 10 monclient: monmap: 2024-03-25T19:37:58.412 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:37:58.412 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:58.412 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:37:58.413237+0000 2024-03-25T19:37:58.412 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:37:58.413237+0000 2024-03-25T19:37:58.412 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:58.412 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:58.412 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:37:58.412 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:37:58.413 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:37:58.413 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:58.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.412+0000 7f821e827640 1 Processor -- start 2024-03-25T19:37:58.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.413+0000 7f821e827640 1 -- start start 2024-03-25T19:37:58.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.413+0000 7f821e827640 10 monclient: init 2024-03-25T19:37:58.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.413+0000 7f821e827640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:58.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.413+0000 7f821e827640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:37:58.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.413+0000 7f821e827640 10 monclient(hunting): picked mon.noname-b con 0x7f82180b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T19:37:58.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.413+0000 7f821e827640 10 monclient(hunting): picked mon.noname-a con 0x7f82180b1730 addr v1:172.21.15.44:6789/0 2024-03-25T19:37:58.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.413+0000 7f821e827640 10 monclient(hunting): picked mon.noname-c con 0x7f821814be90 addr v1:172.21.15.111:6789/0 2024-03-25T19:37:58.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.413+0000 7f821e827640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82180af940 con 0x7f82180b1730 2024-03-25T19:37:58.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.414+0000 7f821e827640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82180b0a70 con 0x7f821814be90 2024-03-25T19:37:58.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.414+0000 7f821e827640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82180b0d50 con 0x7f82180b4d60 2024-03-25T19:37:58.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.414+0000 7f821e827640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:58.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.414+0000 7f821e827640 10 monclient(hunting): authenticate will time out at 5542.584918s 2024-03-25T19:37:58.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.414+0000 7f821cdc4640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f821814be90 0x7f821814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:46402/0 (socket says 172.21.15.44:46402) 2024-03-25T19:37:58.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.414+0000 7f821cdc4640 1 -- 172.21.15.44:0/2934561040 learned_addr learned my addr 172.21.15.44:0/2934561040 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:37:58.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.502+0000 7f8216ffd640 1 -- 172.21.15.44:0/2934561040 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720304733 0 0) 0x7f82180af940 con 0x7f82180b1730 2024-03-25T19:37:58.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.502+0000 7f8216ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:58.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.502+0000 7f8216ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:58.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.502+0000 7f8216ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:58.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.502+0000 7f8216ffd640 10 monclient(hunting): my global_id is 4347 2024-03-25T19:37:58.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.502+0000 7f8216ffd640 1 -- 172.21.15.44:0/2934561040 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8200003520 con 0x7f82180b1730 2024-03-25T19:37:58.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.520+0000 7f8216ffd640 1 -- 172.21.15.44:0/2934561040 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 552541781 0 0) 0x7f8200003520 con 0x7f82180b1730 2024-03-25T19:37:58.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.520+0000 7f8216ffd640 1 -- 172.21.15.44:0/2934561040 >> v1:172.21.15.44:6790/0 conn(0x7f82180b4d60 legacy=0x7f821813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:58.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.520+0000 7f8216ffd640 1 -- 172.21.15.44:0/2934561040 >> v1:172.21.15.111:6789/0 conn(0x7f821814be90 legacy=0x7f821814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:58.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.520+0000 7f8216ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:37:58.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.520+0000 7f8216ffd640 1 monclient: found mon.noname-a 2024-03-25T19:37:58.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.520+0000 7f8216ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:58.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.520+0000 7f8216ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:37:58.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.520+0000 7f8216ffd640 1 -- 172.21.15.44:0/2934561040 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82180b1030 con 0x7f82180b1730 2024-03-25T19:37:58.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.520+0000 7f8216ffd640 10 monclient: _finish_auth 0 2024-03-25T19:37:58.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.520+0000 7f8216ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:37:58.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.520+0000 7f8216ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:58.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.520+0000 7f8216ffd640 1 -- 172.21.15.44:0/2934561040 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f820c001690 con 0x7f82180b1730 2024-03-25T19:37:58.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.520+0000 7f821e827640 5 monclient: authenticate success, global_id 4347 2024-03-25T19:37:58.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.520+0000 7f821e827640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:58.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.520+0000 7f8216ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:58.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.520+0000 7f8216ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:58.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.520+0000 7f8216ffd640 10 monclient: dump: 2024-03-25T19:37:58.521 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:58.521 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:58.521 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:58.521 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:58.521 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:58.521 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:58.521 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:58.521 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:58.522 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:58.522 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:58.522 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.520+0000 7f821e827640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:58.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.538+0000 7f8216ffd640 1 -- 172.21.15.44:0/2934561040 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f820c0017f0 con 0x7f82180b1730 2024-03-25T19:37:58.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.538+0000 7f8216ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:58.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.538+0000 7f8216ffd640 1 -- 172.21.15.44:0/2934561040 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f820c001c40 con 0x7f82180b1730 2024-03-25T19:37:58.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.538+0000 7f821e827640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:58.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.538+0000 7f821e827640 4 set_mon_vals no callback set 2024-03-25T19:37:58.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.538+0000 7f8216ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:58.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.538+0000 7f8216ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:58.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.538+0000 7f8216ffd640 10 monclient: dump: 2024-03-25T19:37:58.539 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:58.539 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:58.539 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:58.539 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:58.539 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:58.539 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:58.539 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:58.539 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:58.539 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:58.539 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:58.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.539+0000 7f821e827640 10 monclient: shutdown 2024-03-25T19:37:58.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.539+0000 7f821e827640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:58.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.539+0000 7f821e827640 1 -- 172.21.15.44:0/2934561040 >> v1:172.21.15.44:6789/0 conn(0x7f82180b1730 legacy=0x7f82180b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:58.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.539+0000 7f821e827640 1 -- 172.21.15.44:0/2934561040 shutdown_connections 2024-03-25T19:37:58.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.539+0000 7f821e827640 1 -- 172.21.15.44:0/2934561040 >> 172.21.15.44:0/2934561040 conn(0x7f82180617a0 msgr2=0x7f8218061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:58.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.539+0000 7f821e827640 1 -- 172.21.15.44:0/2934561040 shutdown_connections 2024-03-25T19:37:58.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.539+0000 7f821e827640 1 -- 172.21.15.44:0/2934561040 wait complete. 2024-03-25T19:37:58.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.539+0000 7f821e827640 10 monclient: build_initial_monmap 2024-03-25T19:37:58.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.539+0000 7f821e827640 10 monclient: monmap: 2024-03-25T19:37:58.542 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:37:58.542 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:58.542 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:37:58.542 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:37:58.542 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:58.542 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:37:58.542 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:37:58.542 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:37:58.542 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:37:58.542 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:58.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.540+0000 7f821e827640 1 librados: starting msgr at 2024-03-25T19:37:58.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.540+0000 7f821e827640 1 librados: starting objecter 2024-03-25T19:37:58.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.540+0000 7f821e827640 1 Processor -- start 2024-03-25T19:37:58.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.540+0000 7f821e827640 1 -- start start 2024-03-25T19:37:58.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.540+0000 7f821e827640 1 librados: setting wanted keys 2024-03-25T19:37:58.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.540+0000 7f821e827640 1 librados: calling monclient init 2024-03-25T19:37:58.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.540+0000 7f821e827640 10 monclient: init 2024-03-25T19:37:58.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.541+0000 7f821e827640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:58.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.541+0000 7f821e827640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:37:58.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.541+0000 7f821e827640 10 monclient(hunting): picked mon.noname-b con 0x7f82180b1730 addr v1:172.21.15.111:6789/0 2024-03-25T19:37:58.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.541+0000 7f821e827640 10 monclient(hunting): picked mon.noname-c con 0x7f82180b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T19:37:58.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.541+0000 7f821e827640 10 monclient(hunting): picked mon.noname-a con 0x7f821814be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:37:58.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.541+0000 7f821e827640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82181420d0 con 0x7f821814be90 2024-03-25T19:37:58.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.541+0000 7f821e827640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8218138270 con 0x7f82180b1730 2024-03-25T19:37:58.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.541+0000 7f821e827640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8218138400 con 0x7f82180b4d60 2024-03-25T19:37:58.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.541+0000 7f821e827640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:58.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.541+0000 7f821e827640 10 monclient(hunting): authenticate will time out at 5542.712276s 2024-03-25T19:37:58.549 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.550+0000 7f8217fff640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f82180b1730 0x7f82181db1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:46452/0 (socket says 172.21.15.44:46452) 2024-03-25T19:37:58.550 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.550+0000 7f8217fff640 1 -- 172.21.15.44:0/3380436840 learned_addr learned my addr 172.21.15.44:0/3380436840 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:37:58.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.551+0000 7f8214ff9640 10 client.?.objecter ms_handle_connect 0x7f82180b1730 2024-03-25T19:37:58.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.551+0000 7f8214ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:58.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.552+0000 7f8214ff9640 10 client.?.objecter ms_handle_connect 0x7f821814be90 2024-03-25T19:37:58.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.552+0000 7f8214ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:58.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.553+0000 7f8214ff9640 10 client.?.objecter ms_handle_connect 0x7f82180b4d60 2024-03-25T19:37:58.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.553+0000 7f8214ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:58.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.556+0000 7f8214ff9640 1 -- 172.21.15.44:0/3380436840 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201935278 0 0) 0x7f8218138270 con 0x7f82180b1730 2024-03-25T19:37:58.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.556+0000 7f8214ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:37:58.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.556+0000 7f8214ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:58.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.556+0000 7f8214ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:58.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.556+0000 7f8214ff9640 10 monclient(hunting): my global_id is 4339 2024-03-25T19:37:58.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.556+0000 7f8214ff9640 1 -- 172.21.15.44:0/3380436840 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8204004eb0 con 0x7f82180b1730 2024-03-25T19:37:58.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.561+0000 7f8214ff9640 1 -- 172.21.15.44:0/3380436840 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035993035 0 0) 0x7f82181420d0 con 0x7f821814be90 2024-03-25T19:37:58.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.561+0000 7f8214ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:37:58.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.561+0000 7f8214ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:58.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.561+0000 7f8214ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:58.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.561+0000 7f8214ff9640 10 monclient(hunting): my global_id is 4350 2024-03-25T19:37:58.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.561+0000 7f8214ff9640 1 -- 172.21.15.44:0/3380436840 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8218138270 con 0x7f821814be90 2024-03-25T19:37:58.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.562+0000 7f8214ff9640 1 -- 172.21.15.44:0/3380436840 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1575634592 0 0) 0x7f8218138400 con 0x7f82180b4d60 2024-03-25T19:37:58.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.562+0000 7f8214ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:37:58.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.562+0000 7f8214ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:58.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.562+0000 7f8214ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:58.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.562+0000 7f8214ff9640 10 monclient(hunting): my global_id is 4322 2024-03-25T19:37:58.562 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.562+0000 7f8214ff9640 1 -- 172.21.15.44:0/3380436840 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82181420d0 con 0x7f82180b4d60 2024-03-25T19:37:58.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.569+0000 7f8214ff9640 1 -- 172.21.15.44:0/3380436840 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4229507480 0 0) 0x7f8204004eb0 con 0x7f82180b1730 2024-03-25T19:37:58.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.569+0000 7f8214ff9640 1 -- 172.21.15.44:0/3380436840 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8218138400 con 0x7f82180b1730 2024-03-25T19:37:58.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.569+0000 7f8214ff9640 1 -- 172.21.15.44:0/3380436840 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f82080019d0 con 0x7f82180b1730 2024-03-25T19:37:58.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.569+0000 7f8214ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:58.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.569+0000 7f8214ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:58.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.569+0000 7f8214ff9640 10 monclient(hunting): dump: 2024-03-25T19:37:58.570 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:58.570 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:58.570 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:58.570 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:58.570 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:58.570 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:58.570 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:58.570 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:58.570 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:58.570 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:58.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.569+0000 7f8214ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:58.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.578+0000 7f8214ff9640 1 -- 172.21.15.44:0/3380436840 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1489524515 0 0) 0x7f8218138400 con 0x7f82180b1730 2024-03-25T19:37:58.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.578+0000 7f8214ff9640 1 -- 172.21.15.44:0/3380436840 >> v1:172.21.15.44:6790/0 conn(0x7f82180b4d60 legacy=0x7f82181405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:58.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.578+0000 7f8214ff9640 1 -- 172.21.15.44:0/3380436840 >> v1:172.21.15.44:6789/0 conn(0x7f821814be90 legacy=0x7f8218134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:58.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.578+0000 7f8214ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:37:58.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.578+0000 7f8214ff9640 1 monclient: found mon.b 2024-03-25T19:37:58.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.578+0000 7f8214ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:58.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.578+0000 7f8214ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:58.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.578+0000 7f8214ff9640 1 -- 172.21.15.44:0/3380436840 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8218142480 con 0x7f82180b1730 2024-03-25T19:37:58.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.578+0000 7f8214ff9640 10 monclient: _finish_auth 0 2024-03-25T19:37:58.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.578+0000 7f8214ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:37:58.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.578+0000 7f8214ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:58.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.578+0000 7f821e827640 5 monclient: authenticate success, global_id 4339 2024-03-25T19:37:58.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.578+0000 7f821e827640 10 monclient: _renew_subs 2024-03-25T19:37:58.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.578+0000 7f821e827640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:58.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.578+0000 7f821e827640 1 -- 172.21.15.44:0/3380436840 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82181426d0 con 0x7f82180b1730 2024-03-25T19:37:58.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.578+0000 7f821e827640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:58.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.578+0000 7f821e827640 10 monclient: _renew_subs 2024-03-25T19:37:58.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.578+0000 7f821e827640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:58.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.578+0000 7f821e827640 1 -- 172.21.15.44:0/3380436840 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82181dbae0 con 0x7f82180b1730 2024-03-25T19:37:58.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.578+0000 7f821e827640 1 librados: init done 2024-03-25T19:37:58.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.580+0000 7f821e827640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:58.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.580+0000 7f821e827640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:58.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.580+0000 7f821e827640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:58.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.580+0000 7f821e827640 1 -- 172.21.15.44:0/3380436840 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82181401f0 con 0x7f82180b1730 2024-03-25T19:37:58.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.854+0000 7f8214ff9640 1 -- 172.21.15.44:0/3380436840 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8208003610 con 0x7f82180b1730 2024-03-25T19:37:58.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.854+0000 7f8214ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:58.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.854+0000 7f8214ff9640 1 -- 172.21.15.44:0/3380436840 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f82080039a0 con 0x7f82180b1730 2024-03-25T19:37:58.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.854+0000 7f8214ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:58.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.854+0000 7f8214ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:58.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.854+0000 7f8214ff9640 10 monclient: dump: 2024-03-25T19:37:58.855 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:37:58.855 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:37:58.855 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:58.855 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:37:58.855 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:58.855 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:37:58.855 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:37:58.855 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:37:58.855 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:37:58.855 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:37:58.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.854+0000 7f82167fc640 4 set_mon_vals no callback set 2024-03-25T19:37:58.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.855+0000 7f8214ff9640 1 -- 172.21.15.44:0/3380436840 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f8208003bc0 con 0x7f82180b1730 2024-03-25T19:37:58.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.856+0000 7f82177fe640 10 monclient: get_auth_request con 0x7f820406dc00 auth_method 0 2024-03-25T19:37:58.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.856+0000 7f8214ff9640 10 client.4339.objecter ms_handle_connect 0x7f820406dc00 2024-03-25T19:37:58.861 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.860+0000 7f8214ff9640 1 -- 172.21.15.44:0/3380436840 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f82080547b0 con 0x7f82180b1730 2024-03-25T19:37:58.861 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.860+0000 7f8214ff9640 10 client.4339.objecter ms_dispatch 0x7f8218000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:58.861 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.860+0000 7f8214ff9640 3 client.4339.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:58.861 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.860+0000 7f8214ff9640 3 client.4339.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:58.861 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.860+0000 7f8214ff9640 20 client.4339.objecter dump_active .. 0 homeless 2024-03-25T19:37:58.861 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.860+0000 7f8214ff9640 1 -- 172.21.15.44:0/3380436840 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f82080502c0 con 0x7f82180b1730 2024-03-25T19:37:58.861 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.860+0000 7f8214ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:58.861 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:58.860+0000 7f8214ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:59.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:59.104+0000 7f821e827640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:59.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:59.104+0000 7f821e827640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:59.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:59.104+0000 7f821e827640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:37:59.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:59.104+0000 7f821e827640 1 -- 172.21.15.44:0/3380436840 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82180627b0 con 0x7f82180b1730 2024-03-25T19:37:59.136 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:59.136+0000 7f8214ff9640 1 -- 172.21.15.44:0/3380436840 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f82080521f0 con 0x7f82180b1730 2024-03-25T19:37:59.136 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:59.136+0000 7f8214ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:59.136 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:59.136+0000 7f8214ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:59.136 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:37:59.136 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:59.138 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:59.138+0000 7f821e827640 10 librados: watch_flush enter 2024-03-25T19:37:59.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:59.138+0000 7f821e827640 10 librados: watch_flush exit 2024-03-25T19:37:59.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:59.138+0000 7f821e827640 20 client.4339.objecter shutdown clearing up homeless session... 2024-03-25T19:37:59.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:59.138+0000 7f821e827640 10 client.4339.objecter successfully canceled tick 2024-03-25T19:37:59.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:59.138+0000 7f821e827640 1 -- 172.21.15.44:0/3380436840 >> v1:172.21.15.111:6800/3883215793 conn(0x7f820406dc00 legacy=0x7f82040700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:59.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:59.138+0000 7f821e827640 10 monclient: shutdown 2024-03-25T19:37:59.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:59.138+0000 7f821e827640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:59.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:59.138+0000 7f821e827640 1 -- 172.21.15.44:0/3380436840 >> v1:172.21.15.111:6789/0 conn(0x7f82180b1730 legacy=0x7f82181db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:59.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:59.138+0000 7f821e827640 1 -- 172.21.15.44:0/3380436840 shutdown_connections 2024-03-25T19:37:59.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:59.138+0000 7f821e827640 1 -- 172.21.15.44:0/3380436840 >> 172.21.15.44:0/3380436840 conn(0x7f82180617a0 msgr2=0x7f82180b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:59.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:59.139+0000 7f821e827640 1 -- 172.21.15.44:0/3380436840 shutdown_connections 2024-03-25T19:37:59.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:59.139+0000 7f821e827640 1 -- 172.21.15.44:0/3380436840 wait complete. 2024-03-25T19:37:59.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:37:59.139+0000 7f821e827640 1 librados: shutdown 2024-03-25T19:37:59.150 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:05.956 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:06.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.073+0000 7fd11977b640 10 monclient: get_monmap_and_config 2024-03-25T19:38:06.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.073+0000 7fd11977b640 10 monclient: build_initial_monmap 2024-03-25T19:38:06.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.073+0000 7fd11977b640 10 monclient: monmap: 2024-03-25T19:38:06.074 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:38:06.074 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:06.074 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:38:06.074970+0000 2024-03-25T19:38:06.074 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:38:06.074970+0000 2024-03-25T19:38:06.074 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:06.074 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:06.074 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:38:06.074 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:38:06.074 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:38:06.074 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:06.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.074+0000 7fd11977b640 1 Processor -- start 2024-03-25T19:38:06.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.074+0000 7fd11977b640 1 -- start start 2024-03-25T19:38:06.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.074+0000 7fd11977b640 10 monclient: init 2024-03-25T19:38:06.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.075+0000 7fd11977b640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:06.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.075+0000 7fd11977b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:38:06.076 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.075+0000 7fd11977b640 10 monclient(hunting): picked mon.noname-b con 0x7fd1140a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T19:38:06.076 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.075+0000 7fd11977b640 10 monclient(hunting): picked mon.noname-c con 0x7fd11413f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:38:06.076 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.075+0000 7fd11977b640 10 monclient(hunting): picked mon.noname-a con 0x7fd1141430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:38:06.076 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.075+0000 7fd11977b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd114142520 con 0x7fd1141430c0 2024-03-25T19:38:06.076 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.075+0000 7fd11977b640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1140b28a0 con 0x7fd11413f710 2024-03-25T19:38:06.076 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.075+0000 7fd11977b640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1140b2a30 con 0x7fd1140a98f0 2024-03-25T19:38:06.076 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.075+0000 7fd11977b640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:06.076 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.075+0000 7fd11977b640 10 monclient(hunting): authenticate will time out at 5550.246768s 2024-03-25T19:38:06.080 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.081+0000 7fd112ffd640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fd1140a98f0 0x7fd1140a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:50812/0 (socket says 172.21.15.44:50812) 2024-03-25T19:38:06.081 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.081+0000 7fd112ffd640 1 -- 172.21.15.44:0/3367258444 learned_addr learned my addr 172.21.15.44:0/3367258444 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:38:06.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.087+0000 7fd1127fc640 1 -- 172.21.15.44:0/3367258444 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317269595 0 0) 0x7fd1140b2a30 con 0x7fd1140a98f0 2024-03-25T19:38:06.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.087+0000 7fd1127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:06.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.087+0000 7fd1127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:06.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.087+0000 7fd1127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:06.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.087+0000 7fd1127fc640 10 monclient(hunting): my global_id is 4325 2024-03-25T19:38:06.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.087+0000 7fd1127fc640 1 -- 172.21.15.44:0/3367258444 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd100003520 con 0x7fd1140a98f0 2024-03-25T19:38:06.092 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.093+0000 7fd1127fc640 1 -- 172.21.15.44:0/3367258444 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2682048193 0 0) 0x7fd114142520 con 0x7fd1141430c0 2024-03-25T19:38:06.093 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.093+0000 7fd1127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:06.093 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.093+0000 7fd1127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:06.093 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.093+0000 7fd1127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:06.093 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.093+0000 7fd1127fc640 10 monclient(hunting): my global_id is 4353 2024-03-25T19:38:06.093 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.093+0000 7fd1127fc640 1 -- 172.21.15.44:0/3367258444 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1140b2a30 con 0x7fd1141430c0 2024-03-25T19:38:06.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.096+0000 7fd1127fc640 1 -- 172.21.15.44:0/3367258444 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035468028 0 0) 0x7fd1140b28a0 con 0x7fd11413f710 2024-03-25T19:38:06.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.096+0000 7fd1127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:06.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.096+0000 7fd1127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:06.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.096+0000 7fd1127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:06.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.096+0000 7fd1127fc640 10 monclient(hunting): my global_id is 4342 2024-03-25T19:38:06.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.096+0000 7fd1127fc640 1 -- 172.21.15.44:0/3367258444 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd114142520 con 0x7fd11413f710 2024-03-25T19:38:06.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.111+0000 7fd1127fc640 1 -- 172.21.15.44:0/3367258444 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2435761234 0 0) 0x7fd1140b2a30 con 0x7fd1141430c0 2024-03-25T19:38:06.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.111+0000 7fd1127fc640 1 -- 172.21.15.44:0/3367258444 >> v1:172.21.15.44:6790/0 conn(0x7fd1140a98f0 legacy=0x7fd1140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:06.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.111+0000 7fd1127fc640 1 -- 172.21.15.44:0/3367258444 >> v1:172.21.15.111:6789/0 conn(0x7fd11413f710 legacy=0x7fd1141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:06.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.111+0000 7fd1127fc640 10 monclient: _finish_hunting 0 2024-03-25T19:38:06.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.111+0000 7fd1127fc640 1 monclient: found mon.noname-a 2024-03-25T19:38:06.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.111+0000 7fd1127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:06.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.111+0000 7fd1127fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:38:06.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.111+0000 7fd1127fc640 1 -- 172.21.15.44:0/3367258444 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1140b44c0 con 0x7fd1141430c0 2024-03-25T19:38:06.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.111+0000 7fd1127fc640 10 monclient: _finish_auth 0 2024-03-25T19:38:06.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.111+0000 7fd1127fc640 10 monclient: _check_auth_tickets 2024-03-25T19:38:06.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.111+0000 7fd1127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:06.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.111+0000 7fd1127fc640 1 -- 172.21.15.44:0/3367258444 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fd10c0019f0 con 0x7fd1141430c0 2024-03-25T19:38:06.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.111+0000 7fd1127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:06.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.111+0000 7fd1127fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:38:06.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.111+0000 7fd1127fc640 10 monclient: dump: 2024-03-25T19:38:06.113 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:06.113 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:06.113 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:06.113 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:06.113 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:06.113 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:06.113 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:06.113 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:06.113 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:06.113 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:06.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.111+0000 7fd11977b640 5 monclient: authenticate success, global_id 4353 2024-03-25T19:38:06.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.111+0000 7fd11977b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:06.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.147+0000 7fd1127fc640 1 -- 172.21.15.44:0/3367258444 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd10c001b50 con 0x7fd1141430c0 2024-03-25T19:38:06.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.147+0000 7fd1127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:06.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.147+0000 7fd1127fc640 1 -- 172.21.15.44:0/3367258444 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fd10c001f60 con 0x7fd1141430c0 2024-03-25T19:38:06.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.147+0000 7fd11977b640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:06.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.147+0000 7fd11977b640 4 set_mon_vals no callback set 2024-03-25T19:38:06.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.147+0000 7fd1127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:06.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.148+0000 7fd1127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:06.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.148+0000 7fd1127fc640 10 monclient: dump: 2024-03-25T19:38:06.148 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:06.148 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:06.148 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:06.148 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:06.148 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:06.148 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:06.148 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:06.148 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:06.148 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:06.148 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:06.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.148+0000 7fd11977b640 10 monclient: shutdown 2024-03-25T19:38:06.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.148+0000 7fd11977b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:06.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.148+0000 7fd11977b640 1 -- 172.21.15.44:0/3367258444 >> v1:172.21.15.44:6789/0 conn(0x7fd1141430c0 legacy=0x7fd11414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:06.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.148+0000 7fd11977b640 1 -- 172.21.15.44:0/3367258444 shutdown_connections 2024-03-25T19:38:06.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.148+0000 7fd11977b640 1 -- 172.21.15.44:0/3367258444 >> 172.21.15.44:0/3367258444 conn(0x7fd1140b7ea0 msgr2=0x7fd1140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:06.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.149+0000 7fd11977b640 1 -- 172.21.15.44:0/3367258444 shutdown_connections 2024-03-25T19:38:06.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.149+0000 7fd11977b640 1 -- 172.21.15.44:0/3367258444 wait complete. 2024-03-25T19:38:06.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.149+0000 7fd11977b640 10 monclient: build_initial_monmap 2024-03-25T19:38:06.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.149+0000 7fd11977b640 10 monclient: monmap: 2024-03-25T19:38:06.150 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:38:06.150 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:06.151 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:38:06.151 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:38:06.151 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:06.151 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:38:06.151 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:38:06.151 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:38:06.151 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:38:06.151 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:06.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.150+0000 7fd11977b640 1 librados: starting msgr at 2024-03-25T19:38:06.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.150+0000 7fd11977b640 1 librados: starting objecter 2024-03-25T19:38:06.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.150+0000 7fd11977b640 1 Processor -- start 2024-03-25T19:38:06.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.150+0000 7fd11977b640 1 -- start start 2024-03-25T19:38:06.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.150+0000 7fd11977b640 1 librados: setting wanted keys 2024-03-25T19:38:06.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.150+0000 7fd11977b640 1 librados: calling monclient init 2024-03-25T19:38:06.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.150+0000 7fd11977b640 10 monclient: init 2024-03-25T19:38:06.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.151+0000 7fd11977b640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:06.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.151+0000 7fd11977b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:38:06.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.151+0000 7fd11977b640 10 monclient(hunting): picked mon.noname-b con 0x7fd1140a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:38:06.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.151+0000 7fd11977b640 10 monclient(hunting): picked mon.noname-a con 0x7fd11413f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:38:06.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.151+0000 7fd11977b640 10 monclient(hunting): picked mon.noname-c con 0x7fd1141430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:38:06.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.151+0000 7fd11977b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd11406f460 con 0x7fd11413f710 2024-03-25T19:38:06.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.151+0000 7fd11977b640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1140701c0 con 0x7fd1140a98f0 2024-03-25T19:38:06.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.151+0000 7fd11977b640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd11406f180 con 0x7fd1141430c0 2024-03-25T19:38:06.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.151+0000 7fd11977b640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:06.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.151+0000 7fd11977b640 10 monclient(hunting): authenticate will time out at 5550.322590s 2024-03-25T19:38:06.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.166+0000 7fd10bfff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fd11413f710 0x7fd114070930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57146/0 (socket says 172.21.15.44:57146) 2024-03-25T19:38:06.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.166+0000 7fd10bfff640 1 -- 172.21.15.44:0/865180574 learned_addr learned my addr 172.21.15.44:0/865180574 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:38:06.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.167+0000 7fd10b7fe640 10 client.?.objecter ms_handle_connect 0x7fd11413f710 2024-03-25T19:38:06.167 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.167+0000 7fd10b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:06.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.169+0000 7fd10b7fe640 10 client.?.objecter ms_handle_connect 0x7fd1141430c0 2024-03-25T19:38:06.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.169+0000 7fd10b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:06.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.170+0000 7fd10b7fe640 10 client.?.objecter ms_handle_connect 0x7fd1140a98f0 2024-03-25T19:38:06.170 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.170+0000 7fd10b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:06.176 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.177+0000 7fd10b7fe640 1 -- 172.21.15.44:0/865180574 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267390536 0 0) 0x7fd1140701c0 con 0x7fd1140a98f0 2024-03-25T19:38:06.176 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.177+0000 7fd10b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:06.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.177+0000 7fd10b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:06.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.177+0000 7fd10b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:06.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.177+0000 7fd10b7fe640 10 monclient(hunting): my global_id is 4345 2024-03-25T19:38:06.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.177+0000 7fd10b7fe640 1 -- 172.21.15.44:0/865180574 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0f4004eb0 con 0x7fd1140a98f0 2024-03-25T19:38:06.178 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.178+0000 7fd10b7fe640 1 -- 172.21.15.44:0/865180574 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717601534 0 0) 0x7fd11406f180 con 0x7fd1141430c0 2024-03-25T19:38:06.178 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.178+0000 7fd10b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:06.178 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.178+0000 7fd10b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:06.178 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.178+0000 7fd10b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:06.178 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.178+0000 7fd10b7fe640 10 monclient(hunting): my global_id is 4328 2024-03-25T19:38:06.179 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.178+0000 7fd10b7fe640 1 -- 172.21.15.44:0/865180574 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1140701c0 con 0x7fd1141430c0 2024-03-25T19:38:06.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.189+0000 7fd10b7fe640 1 -- 172.21.15.44:0/865180574 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1063192570 0 0) 0x7fd0f4004eb0 con 0x7fd1140a98f0 2024-03-25T19:38:06.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.189+0000 7fd10b7fe640 1 -- 172.21.15.44:0/865180574 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd11406f180 con 0x7fd1140a98f0 2024-03-25T19:38:06.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.189+0000 7fd10b7fe640 1 -- 172.21.15.44:0/865180574 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fd104001950 con 0x7fd1140a98f0 2024-03-25T19:38:06.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.189+0000 7fd10b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:06.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.189+0000 7fd10b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:06.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.189+0000 7fd10b7fe640 10 monclient(hunting): dump: 2024-03-25T19:38:06.189 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:06.189 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:06.189 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:06.189 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:06.189 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:06.189 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:06.189 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:06.190 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:06.190 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:06.190 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:06.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.189+0000 7fd10b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:06.193 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.193+0000 7fd10b7fe640 1 -- 172.21.15.44:0/865180574 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3366789022 0 0) 0x7fd1140701c0 con 0x7fd1141430c0 2024-03-25T19:38:06.193 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.193+0000 7fd10b7fe640 1 -- 172.21.15.44:0/865180574 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd0f4004eb0 con 0x7fd1141430c0 2024-03-25T19:38:06.193 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.193+0000 7fd10b7fe640 1 -- 172.21.15.44:0/865180574 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fd10c001e60 con 0x7fd1141430c0 2024-03-25T19:38:06.193 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.193+0000 7fd10b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:06.193 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.193+0000 7fd10b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:06.193 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.193+0000 7fd10b7fe640 10 monclient(hunting): dump: 2024-03-25T19:38:06.193 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:06.194 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:06.194 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:06.194 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:06.194 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:06.194 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:06.194 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:06.194 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:06.194 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:06.194 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:06.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.196+0000 7fd10b7fe640 1 -- 172.21.15.44:0/865180574 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 530727089 0 0) 0x7fd11406f460 con 0x7fd11413f710 2024-03-25T19:38:06.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.196+0000 7fd10b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:06.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.196+0000 7fd10b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:06.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.196+0000 7fd10b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:06.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.196+0000 7fd10b7fe640 10 monclient(hunting): my global_id is 4356 2024-03-25T19:38:06.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.196+0000 7fd10b7fe640 1 -- 172.21.15.44:0/865180574 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1140701c0 con 0x7fd11413f710 2024-03-25T19:38:06.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.201+0000 7fd10b7fe640 1 -- 172.21.15.44:0/865180574 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1323840215 0 0) 0x7fd11406f180 con 0x7fd1140a98f0 2024-03-25T19:38:06.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.201+0000 7fd10b7fe640 1 -- 172.21.15.44:0/865180574 >> v1:172.21.15.44:6790/0 conn(0x7fd1141430c0 legacy=0x7fd1141eebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:06.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.201+0000 7fd10b7fe640 1 -- 172.21.15.44:0/865180574 >> v1:172.21.15.44:6789/0 conn(0x7fd11413f710 legacy=0x7fd114070930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:06.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.201+0000 7fd10b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:38:06.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.201+0000 7fd10b7fe640 1 monclient: found mon.b 2024-03-25T19:38:06.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.201+0000 7fd10b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:06.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.201+0000 7fd10b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:06.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.201+0000 7fd10b7fe640 1 -- 172.21.15.44:0/865180574 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11406e280 con 0x7fd1140a98f0 2024-03-25T19:38:06.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.201+0000 7fd10b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:38:06.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.201+0000 7fd10b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:38:06.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.201+0000 7fd10b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:06.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.201+0000 7fd11977b640 5 monclient: authenticate success, global_id 4345 2024-03-25T19:38:06.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.201+0000 7fd11977b640 10 monclient: _renew_subs 2024-03-25T19:38:06.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.201+0000 7fd11977b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:06.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.201+0000 7fd11977b640 1 -- 172.21.15.44:0/865180574 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd114071900 con 0x7fd1140a98f0 2024-03-25T19:38:06.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.201+0000 7fd11977b640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:06.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.201+0000 7fd11977b640 10 monclient: _renew_subs 2024-03-25T19:38:06.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.201+0000 7fd11977b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:06.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.201+0000 7fd11977b640 1 -- 172.21.15.44:0/865180574 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd114071220 con 0x7fd1140a98f0 2024-03-25T19:38:06.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.201+0000 7fd11977b640 1 librados: init done 2024-03-25T19:38:06.203 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.202+0000 7fd11977b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:06.203 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.202+0000 7fd11977b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:06.203 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.202+0000 7fd11977b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:06.203 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.202+0000 7fd11977b640 1 -- 172.21.15.44:0/865180574 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1140627b0 con 0x7fd1140a98f0 2024-03-25T19:38:06.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.456+0000 7fd10b7fe640 1 -- 172.21.15.44:0/865180574 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd104001d60 con 0x7fd1140a98f0 2024-03-25T19:38:06.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.456+0000 7fd10b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:06.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.456+0000 7fd10b7fe640 1 -- 172.21.15.44:0/865180574 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fd104004980 con 0x7fd1140a98f0 2024-03-25T19:38:06.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.456+0000 7fd10b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:06.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.457+0000 7fd10b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:06.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.457+0000 7fd10b7fe640 10 monclient: dump: 2024-03-25T19:38:06.457 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:06.457 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:06.457 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:06.457 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:06.457 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:06.457 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:06.457 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:06.457 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:06.457 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:06.458 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:06.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.456+0000 7fd111ffb640 4 set_mon_vals no callback set 2024-03-25T19:38:06.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.458+0000 7fd10b7fe640 1 -- 172.21.15.44:0/865180574 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7fd10401aa10 con 0x7fd1140a98f0 2024-03-25T19:38:06.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.458+0000 7fd10bfff640 10 monclient: get_auth_request con 0x7fd0f406df50 auth_method 0 2024-03-25T19:38:06.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.459+0000 7fd10b7fe640 10 client.4345.objecter ms_handle_connect 0x7fd0f406df50 2024-03-25T19:38:06.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.464+0000 7fd10b7fe640 1 -- 172.21.15.44:0/865180574 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7fd104054740 con 0x7fd1140a98f0 2024-03-25T19:38:06.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.464+0000 7fd10b7fe640 10 client.4345.objecter ms_dispatch 0x7fd114000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:06.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.464+0000 7fd10b7fe640 3 client.4345.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:06.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.464+0000 7fd10b7fe640 3 client.4345.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:06.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.464+0000 7fd10b7fe640 20 client.4345.objecter dump_active .. 0 homeless 2024-03-25T19:38:06.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.469+0000 7fd10b7fe640 1 -- 172.21.15.44:0/865180574 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd104050080 con 0x7fd1140a98f0 2024-03-25T19:38:06.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.469+0000 7fd10b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:06.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.469+0000 7fd10b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:06.697 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.697+0000 7fd11977b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:06.697 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.697+0000 7fd11977b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:06.697 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.697+0000 7fd11977b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:06.697 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.697+0000 7fd11977b640 1 -- 172.21.15.44:0/865180574 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd114069f40 con 0x7fd1140a98f0 2024-03-25T19:38:06.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.809+0000 7fd10b7fe640 1 -- 172.21.15.44:0/865180574 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7fd104052180 con 0x7fd1140a98f0 2024-03-25T19:38:06.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.809+0000 7fd10b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:06.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.809+0000 7fd10b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:06.809 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:38:06.809 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:06.811 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.811+0000 7fd11977b640 10 librados: watch_flush enter 2024-03-25T19:38:06.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.811+0000 7fd11977b640 10 librados: watch_flush exit 2024-03-25T19:38:06.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.811+0000 7fd11977b640 20 client.4345.objecter shutdown clearing up homeless session... 2024-03-25T19:38:06.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.811+0000 7fd11977b640 10 client.4345.objecter successfully canceled tick 2024-03-25T19:38:06.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.811+0000 7fd11977b640 1 -- 172.21.15.44:0/865180574 >> v1:172.21.15.111:6800/3883215793 conn(0x7fd0f406df50 legacy=0x7fd0f4070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:06.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.811+0000 7fd11977b640 10 monclient: shutdown 2024-03-25T19:38:06.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.811+0000 7fd11977b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:06.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.811+0000 7fd11977b640 1 -- 172.21.15.44:0/865180574 >> v1:172.21.15.111:6789/0 conn(0x7fd1140a98f0 legacy=0x7fd11406ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:06.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.811+0000 7fd11977b640 1 -- 172.21.15.44:0/865180574 shutdown_connections 2024-03-25T19:38:06.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.811+0000 7fd11977b640 1 -- 172.21.15.44:0/865180574 >> 172.21.15.44:0/865180574 conn(0x7fd1140b7ea0 msgr2=0x7fd114074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:06.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.811+0000 7fd11977b640 1 -- 172.21.15.44:0/865180574 shutdown_connections 2024-03-25T19:38:06.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.811+0000 7fd11977b640 1 -- 172.21.15.44:0/865180574 wait complete. 2024-03-25T19:38:06.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:06.811+0000 7fd11977b640 1 librados: shutdown 2024-03-25T19:38:06.824 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:13.630 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:13.748 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.749+0000 7f64b8b2b640 10 monclient: get_monmap_and_config 2024-03-25T19:38:13.748 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.749+0000 7f64b8b2b640 10 monclient: build_initial_monmap 2024-03-25T19:38:13.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.749+0000 7f64b8b2b640 10 monclient: monmap: 2024-03-25T19:38:13.749 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:38:13.749 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:13.749 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:38:13.750229+0000 2024-03-25T19:38:13.749 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:38:13.750229+0000 2024-03-25T19:38:13.749 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:13.749 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:13.749 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:38:13.749 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:38:13.749 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:38:13.749 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:13.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.749+0000 7f64b8b2b640 1 Processor -- start 2024-03-25T19:38:13.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.749+0000 7f64b8b2b640 1 -- start start 2024-03-25T19:38:13.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.749+0000 7f64b8b2b640 10 monclient: init 2024-03-25T19:38:13.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.750+0000 7f64b8b2b640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:13.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.750+0000 7f64b8b2b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:38:13.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.750+0000 7f64b8b2b640 10 monclient(hunting): picked mon.noname-b con 0x7f64b40b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T19:38:13.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.750+0000 7f64b8b2b640 10 monclient(hunting): picked mon.noname-c con 0x7f64b40b1730 addr v1:172.21.15.111:6789/0 2024-03-25T19:38:13.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.750+0000 7f64b8b2b640 10 monclient(hunting): picked mon.noname-a con 0x7f64b414be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:38:13.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.750+0000 7f64b8b2b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b40af940 con 0x7f64b414be90 2024-03-25T19:38:13.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.750+0000 7f64b8b2b640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b40b0a70 con 0x7f64b40b1730 2024-03-25T19:38:13.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.750+0000 7f64b8b2b640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b40b0d50 con 0x7f64b40b4d60 2024-03-25T19:38:13.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.750+0000 7f64b8b2b640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:13.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.750+0000 7f64b8b2b640 10 monclient(hunting): authenticate will time out at 5557.921542s 2024-03-25T19:38:13.758 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.759+0000 7f64b2d9d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f64b414be90 0x7f64b414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:39736/0 (socket says 172.21.15.44:39736) 2024-03-25T19:38:13.758 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.759+0000 7f64b2d9d640 1 -- 172.21.15.44:0/683396815 learned_addr learned my addr 172.21.15.44:0/683396815 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:38:13.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.773+0000 7f64b1d9b640 1 -- 172.21.15.44:0/683396815 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 297213517 0 0) 0x7f64b40af940 con 0x7f64b414be90 2024-03-25T19:38:13.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.773+0000 7f64b1d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:38:13.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.773+0000 7f64b1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:13.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.773+0000 7f64b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:13.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.773+0000 7f64b1d9b640 10 monclient(hunting): my global_id is 4359 2024-03-25T19:38:13.774 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.773+0000 7f64b1d9b640 1 -- 172.21.15.44:0/683396815 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a0003520 con 0x7f64b414be90 2024-03-25T19:38:13.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.780+0000 7f64b1d9b640 1 -- 172.21.15.44:0/683396815 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186832546 0 0) 0x7f64b40b0a70 con 0x7f64b40b1730 2024-03-25T19:38:13.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.780+0000 7f64b1d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:38:13.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.780+0000 7f64b1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:13.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.780+0000 7f64b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:13.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.780+0000 7f64b1d9b640 10 monclient(hunting): my global_id is 4348 2024-03-25T19:38:13.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.780+0000 7f64b1d9b640 1 -- 172.21.15.44:0/683396815 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b40af940 con 0x7f64b40b1730 2024-03-25T19:38:13.784 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.784+0000 7f64b1d9b640 1 -- 172.21.15.44:0/683396815 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 292942504 0 0) 0x7f64b40b0d50 con 0x7f64b40b4d60 2024-03-25T19:38:13.784 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.784+0000 7f64b1d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:38:13.784 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.784+0000 7f64b1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:13.785 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.784+0000 7f64b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:13.785 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.784+0000 7f64b1d9b640 10 monclient(hunting): my global_id is 4331 2024-03-25T19:38:13.785 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.784+0000 7f64b1d9b640 1 -- 172.21.15.44:0/683396815 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b40b0a70 con 0x7f64b40b4d60 2024-03-25T19:38:13.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.794+0000 7f64b1d9b640 1 -- 172.21.15.44:0/683396815 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3757437735 0 0) 0x7f64b40b0a70 con 0x7f64b40b4d60 2024-03-25T19:38:13.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.794+0000 7f64b1d9b640 1 -- 172.21.15.44:0/683396815 >> v1:172.21.15.111:6789/0 conn(0x7f64b40b1730 legacy=0x7f64b40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:13.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.794+0000 7f64b1d9b640 1 -- 172.21.15.44:0/683396815 >> v1:172.21.15.44:6789/0 conn(0x7f64b414be90 legacy=0x7f64b414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:13.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.794+0000 7f64b1d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:38:13.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.794+0000 7f64b1d9b640 1 monclient: found mon.noname-b 2024-03-25T19:38:13.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.794+0000 7f64b1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:13.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.794+0000 7f64b1d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:38:13.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.794+0000 7f64b1d9b640 1 -- 172.21.15.44:0/683396815 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b40b1030 con 0x7f64b40b4d60 2024-03-25T19:38:13.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.794+0000 7f64b1d9b640 10 monclient: _finish_auth 0 2024-03-25T19:38:13.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.794+0000 7f64b1d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:38:13.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.794+0000 7f64b1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:13.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.794+0000 7f64b1d9b640 1 -- 172.21.15.44:0/683396815 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f64a40015b0 con 0x7f64b40b4d60 2024-03-25T19:38:13.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.794+0000 7f64b8b2b640 5 monclient: authenticate success, global_id 4331 2024-03-25T19:38:13.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.794+0000 7f64b8b2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:13.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.794+0000 7f64b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:13.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.794+0000 7f64b1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:13.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.794+0000 7f64b1d9b640 10 monclient: dump: 2024-03-25T19:38:13.796 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:13.796 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:13.796 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:13.796 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:13.796 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:13.796 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:13.796 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:13.796 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:13.796 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:13.796 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:13.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.794+0000 7f64b8b2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:13.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.818+0000 7f64b1d9b640 1 -- 172.21.15.44:0/683396815 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64a4001710 con 0x7f64b40b4d60 2024-03-25T19:38:13.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.818+0000 7f64b1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:13.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.818+0000 7f64b1d9b640 1 -- 172.21.15.44:0/683396815 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f64a4001b60 con 0x7f64b40b4d60 2024-03-25T19:38:13.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.818+0000 7f64b8b2b640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:13.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.818+0000 7f64b8b2b640 4 set_mon_vals no callback set 2024-03-25T19:38:13.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.818+0000 7f64b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:13.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.818+0000 7f64b1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:13.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.818+0000 7f64b1d9b640 10 monclient: dump: 2024-03-25T19:38:13.818 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:13.818 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:13.818 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:13.818 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:13.818 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:13.819 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:13.819 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:13.819 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:13.819 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:13.819 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:13.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.818+0000 7f64b8b2b640 10 monclient: shutdown 2024-03-25T19:38:13.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.818+0000 7f64b8b2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:13.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.818+0000 7f64b8b2b640 1 -- 172.21.15.44:0/683396815 >> v1:172.21.15.44:6790/0 conn(0x7f64b40b4d60 legacy=0x7f64b413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:13.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.818+0000 7f64b8b2b640 1 -- 172.21.15.44:0/683396815 shutdown_connections 2024-03-25T19:38:13.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.818+0000 7f64b8b2b640 1 -- 172.21.15.44:0/683396815 >> 172.21.15.44:0/683396815 conn(0x7f64b40617a0 msgr2=0x7f64b4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:13.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.819+0000 7f64b8b2b640 1 -- 172.21.15.44:0/683396815 shutdown_connections 2024-03-25T19:38:13.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.819+0000 7f64b8b2b640 1 -- 172.21.15.44:0/683396815 wait complete. 2024-03-25T19:38:13.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.819+0000 7f64b8b2b640 10 monclient: build_initial_monmap 2024-03-25T19:38:13.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.819+0000 7f64b8b2b640 10 monclient: monmap: 2024-03-25T19:38:13.822 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:38:13.822 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:13.822 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:38:13.822 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:38:13.823 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:13.823 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:38:13.823 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:38:13.823 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:38:13.823 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:38:13.823 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:13.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.820+0000 7f64b8b2b640 1 librados: starting msgr at 2024-03-25T19:38:13.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.820+0000 7f64b8b2b640 1 librados: starting objecter 2024-03-25T19:38:13.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.820+0000 7f64b8b2b640 1 Processor -- start 2024-03-25T19:38:13.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.820+0000 7f64b8b2b640 1 -- start start 2024-03-25T19:38:13.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.820+0000 7f64b8b2b640 1 librados: setting wanted keys 2024-03-25T19:38:13.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.820+0000 7f64b8b2b640 1 librados: calling monclient init 2024-03-25T19:38:13.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.820+0000 7f64b8b2b640 10 monclient: init 2024-03-25T19:38:13.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.821+0000 7f64b8b2b640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:13.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.821+0000 7f64b8b2b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:38:13.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.821+0000 7f64b8b2b640 10 monclient(hunting): picked mon.noname-b con 0x7f64b40b1730 addr v1:172.21.15.111:6789/0 2024-03-25T19:38:13.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.821+0000 7f64b8b2b640 10 monclient(hunting): picked mon.noname-c con 0x7f64b40b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T19:38:13.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.821+0000 7f64b8b2b640 10 monclient(hunting): picked mon.noname-a con 0x7f64b414be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:38:13.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.821+0000 7f64b8b2b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b4138e00 con 0x7f64b414be90 2024-03-25T19:38:13.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.821+0000 7f64b8b2b640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b4139ef0 con 0x7f64b40b1730 2024-03-25T19:38:13.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.821+0000 7f64b8b2b640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b413a080 con 0x7f64b40b4d60 2024-03-25T19:38:13.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.821+0000 7f64b8b2b640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:13.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.821+0000 7f64b8b2b640 10 monclient(hunting): authenticate will time out at 5557.992189s 2024-03-25T19:38:13.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.823+0000 7f64b259c640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f64b40b1730 0x7f64b413b630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:52708/0 (socket says 172.21.15.44:52708) 2024-03-25T19:38:13.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.823+0000 7f64b259c640 1 -- 172.21.15.44:0/1733251035 learned_addr learned my addr 172.21.15.44:0/1733251035 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:38:13.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.825+0000 7f64aaffd640 10 client.?.objecter ms_handle_connect 0x7f64b40b1730 2024-03-25T19:38:13.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.825+0000 7f64aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:13.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.835+0000 7f64aaffd640 1 -- 172.21.15.44:0/1733251035 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33825502 0 0) 0x7f64b4139ef0 con 0x7f64b40b1730 2024-03-25T19:38:13.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.835+0000 7f64aaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:13.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.835+0000 7f64aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:13.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.835+0000 7f64aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:13.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.835+0000 7f64aaffd640 10 monclient(hunting): my global_id is 4351 2024-03-25T19:38:13.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.835+0000 7f64aaffd640 1 -- 172.21.15.44:0/1733251035 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6494004eb0 con 0x7f64b40b1730 2024-03-25T19:38:13.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.836+0000 7f64aaffd640 10 client.?.objecter ms_handle_connect 0x7f64b40b4d60 2024-03-25T19:38:13.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.836+0000 7f64aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:13.843 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.843+0000 7f64aaffd640 10 client.?.objecter ms_handle_connect 0x7f64b414be90 2024-03-25T19:38:13.843 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.843+0000 7f64aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:13.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.850+0000 7f64aaffd640 1 -- 172.21.15.44:0/1733251035 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3847007912 0 0) 0x7f6494004eb0 con 0x7f64b40b1730 2024-03-25T19:38:13.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.850+0000 7f64aaffd640 1 -- 172.21.15.44:0/1733251035 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64b4139ef0 con 0x7f64b40b1730 2024-03-25T19:38:13.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.850+0000 7f64aaffd640 1 -- 172.21.15.44:0/1733251035 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f64a4001b20 con 0x7f64b40b1730 2024-03-25T19:38:13.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.850+0000 7f64aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:13.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.850+0000 7f64aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:13.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.850+0000 7f64aaffd640 10 monclient(hunting): dump: 2024-03-25T19:38:13.850 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:13.850 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:13.851 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:13.851 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:13.851 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:13.851 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:13.851 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:13.851 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:13.851 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:13.851 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:13.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.850+0000 7f64aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:13.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.857+0000 7f64aaffd640 1 -- 172.21.15.44:0/1733251035 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3972919265 0 0) 0x7f64b4139ef0 con 0x7f64b40b1730 2024-03-25T19:38:13.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.857+0000 7f64aaffd640 1 -- 172.21.15.44:0/1733251035 >> v1:172.21.15.44:6790/0 conn(0x7f64b40b4d60 legacy=0x7f64b41daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:13.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.857+0000 7f64aaffd640 1 -- 172.21.15.44:0/1733251035 >> v1:172.21.15.44:6789/0 conn(0x7f64b414be90 legacy=0x7f64b4132d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:13.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.857+0000 7f64aaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:38:13.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.857+0000 7f64aaffd640 1 monclient: found mon.b 2024-03-25T19:38:13.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.857+0000 7f64aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:13.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.857+0000 7f64aaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:13.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.857+0000 7f64aaffd640 1 -- 172.21.15.44:0/1733251035 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b41452d0 con 0x7f64b40b1730 2024-03-25T19:38:13.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.857+0000 7f64aaffd640 10 monclient: _finish_auth 0 2024-03-25T19:38:13.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.857+0000 7f64aaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:38:13.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.857+0000 7f64aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:13.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.857+0000 7f64b8b2b640 5 monclient: authenticate success, global_id 4351 2024-03-25T19:38:13.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.857+0000 7f64b8b2b640 10 monclient: _renew_subs 2024-03-25T19:38:13.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.857+0000 7f64b8b2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:13.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.857+0000 7f64b8b2b640 1 -- 172.21.15.44:0/1733251035 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b4145580 con 0x7f64b40b1730 2024-03-25T19:38:13.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.857+0000 7f64b8b2b640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:13.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.857+0000 7f64b8b2b640 10 monclient: _renew_subs 2024-03-25T19:38:13.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.857+0000 7f64b8b2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:13.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.857+0000 7f64b8b2b640 1 -- 172.21.15.44:0/1733251035 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b4133610 con 0x7f64b40b1730 2024-03-25T19:38:13.858 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.857+0000 7f64b8b2b640 1 librados: init done 2024-03-25T19:38:13.859 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.858+0000 7f64b8b2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:13.859 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.858+0000 7f64b8b2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:13.859 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.858+0000 7f64b8b2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:13.859 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:13.858+0000 7f64b8b2b640 1 -- 172.21.15.44:0/1733251035 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b4138f90 con 0x7f64b40b1730 2024-03-25T19:38:14.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.134+0000 7f64aaffd640 1 -- 172.21.15.44:0/1733251035 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64a40038a0 con 0x7f64b40b1730 2024-03-25T19:38:14.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.134+0000 7f64aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:14.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.134+0000 7f64aaffd640 1 -- 172.21.15.44:0/1733251035 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f64a4003cb0 con 0x7f64b40b1730 2024-03-25T19:38:14.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.134+0000 7f64aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:14.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.134+0000 7f64aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:14.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.134+0000 7f64aaffd640 10 monclient: dump: 2024-03-25T19:38:14.134 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:14.134 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:14.134 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:14.135 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:14.135 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:14.135 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:14.135 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:14.135 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:14.135 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:14.136 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:14.136 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.134+0000 7f64b159a640 4 set_mon_vals no callback set 2024-03-25T19:38:14.136 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.135+0000 7f64aaffd640 1 -- 172.21.15.44:0/1733251035 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f64a4003ed0 con 0x7f64b40b1730 2024-03-25T19:38:14.136 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.135+0000 7f64abfff640 10 monclient: get_auth_request con 0x7f649406dd00 auth_method 0 2024-03-25T19:38:14.136 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.135+0000 7f64aaffd640 10 client.4351.objecter ms_handle_connect 0x7f649406dd00 2024-03-25T19:38:14.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.145+0000 7f64aaffd640 1 -- 172.21.15.44:0/1733251035 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f64a4054940 con 0x7f64b40b1730 2024-03-25T19:38:14.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.145+0000 7f64aaffd640 10 client.4351.objecter ms_dispatch 0x7f64b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:14.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.145+0000 7f64aaffd640 3 client.4351.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:14.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.145+0000 7f64aaffd640 3 client.4351.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:14.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.145+0000 7f64aaffd640 20 client.4351.objecter dump_active .. 0 homeless 2024-03-25T19:38:14.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.147+0000 7f64aaffd640 1 -- 172.21.15.44:0/1733251035 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f64a4050450 con 0x7f64b40b1730 2024-03-25T19:38:14.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.147+0000 7f64aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:14.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.147+0000 7f64aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:14.268 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:03:05.170 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:14.273 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:03:05.175 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:14.297 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:03:05.199 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:14.326 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:03:05.206 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:14.329 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:03:05.209 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:14.356 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:03:05.258 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:14.358 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:03:05.237 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:14.364 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:03:05.266 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:14.381 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.382+0000 7f64b8b2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:14.382 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.382+0000 7f64b8b2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:14.382 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.382+0000 7f64b8b2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:14.382 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.382+0000 7f64b8b2b640 1 -- 172.21.15.44:0/1733251035 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64b40627b0 con 0x7f64b40b1730 2024-03-25T19:38:14.392 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:03:05.294 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:14.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.425+0000 7f64aaffd640 1 -- 172.21.15.44:0/1733251035 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f64a4052380 con 0x7f64b40b1730 2024-03-25T19:38:14.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.425+0000 7f64aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:14.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.425+0000 7f64aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:14.426 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:38:14.426 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:14.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.427+0000 7f64b8b2b640 10 librados: watch_flush enter 2024-03-25T19:38:14.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.428+0000 7f64b8b2b640 10 librados: watch_flush exit 2024-03-25T19:38:14.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.428+0000 7f64b8b2b640 20 client.4351.objecter shutdown clearing up homeless session... 2024-03-25T19:38:14.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.428+0000 7f64b8b2b640 10 client.4351.objecter successfully canceled tick 2024-03-25T19:38:14.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.428+0000 7f64b8b2b640 1 -- 172.21.15.44:0/1733251035 >> v1:172.21.15.111:6800/3883215793 conn(0x7f649406dd00 legacy=0x7f64940701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:14.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.428+0000 7f64b8b2b640 10 monclient: shutdown 2024-03-25T19:38:14.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.428+0000 7f64b8b2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:14.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.428+0000 7f64b8b2b640 1 -- 172.21.15.44:0/1733251035 >> v1:172.21.15.111:6789/0 conn(0x7f64b40b1730 legacy=0x7f64b413b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:14.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.428+0000 7f64b8b2b640 1 -- 172.21.15.44:0/1733251035 shutdown_connections 2024-03-25T19:38:14.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.428+0000 7f64b8b2b640 1 -- 172.21.15.44:0/1733251035 >> 172.21.15.44:0/1733251035 conn(0x7f64b40617a0 msgr2=0x7f64b40b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:14.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.428+0000 7f64b8b2b640 1 -- 172.21.15.44:0/1733251035 shutdown_connections 2024-03-25T19:38:14.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.428+0000 7f64b8b2b640 1 -- 172.21.15.44:0/1733251035 wait complete. 2024-03-25T19:38:14.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:14.428+0000 7f64b8b2b640 1 librados: shutdown 2024-03-25T19:38:14.439 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:21.247 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:21.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.356+0000 7f246ad3a640 10 monclient: get_monmap_and_config 2024-03-25T19:38:21.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.356+0000 7f246ad3a640 10 monclient: build_initial_monmap 2024-03-25T19:38:21.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.356+0000 7f246ad3a640 10 monclient: monmap: 2024-03-25T19:38:21.357 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:38:21.357 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:21.357 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:38:21.357855+0000 2024-03-25T19:38:21.357 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:38:21.357855+0000 2024-03-25T19:38:21.357 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:21.357 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:21.357 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:38:21.357 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:38:21.357 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:38:21.357 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:21.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.357+0000 7f246ad3a640 1 Processor -- start 2024-03-25T19:38:21.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.357+0000 7f246ad3a640 1 -- start start 2024-03-25T19:38:21.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.357+0000 7f246ad3a640 10 monclient: init 2024-03-25T19:38:21.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.358+0000 7f246ad3a640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:21.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.358+0000 7f246ad3a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:38:21.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.358+0000 7f246ad3a640 10 monclient(hunting): picked mon.noname-a con 0x7f24640a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:38:21.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.358+0000 7f246ad3a640 10 monclient(hunting): picked mon.noname-b con 0x7f246413f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:38:21.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.358+0000 7f246ad3a640 10 monclient(hunting): picked mon.noname-c con 0x7f24641430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:38:21.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.358+0000 7f246ad3a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2464142520 con 0x7f24640a98f0 2024-03-25T19:38:21.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.358+0000 7f246ad3a640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24640b28a0 con 0x7f24641430c0 2024-03-25T19:38:21.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.358+0000 7f246ad3a640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24640b2a30 con 0x7f246413f710 2024-03-25T19:38:21.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.358+0000 7f246ad3a640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:21.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.358+0000 7f246ad3a640 10 monclient(hunting): authenticate will time out at 5565.529066s 2024-03-25T19:38:21.366 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.367+0000 7f2463fff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f246413f710 0x7f24641429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:48370/0 (socket says 172.21.15.44:48370) 2024-03-25T19:38:21.366 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.367+0000 7f2463fff640 1 -- 172.21.15.44:0/1074009116 learned_addr learned my addr 172.21.15.44:0/1074009116 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:38:21.373 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.373+0000 7f24637fe640 1 -- 172.21.15.44:0/1074009116 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233232472 0 0) 0x7f24640b2a30 con 0x7f246413f710 2024-03-25T19:38:21.373 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.374+0000 7f24637fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:21.373 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.374+0000 7f24637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:21.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.374+0000 7f24637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:21.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.374+0000 7f24637fe640 10 monclient(hunting): my global_id is 4337 2024-03-25T19:38:21.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.374+0000 7f24637fe640 1 -- 172.21.15.44:0/1074009116 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2450003520 con 0x7f246413f710 2024-03-25T19:38:21.385 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.385+0000 7f24637fe640 1 -- 172.21.15.44:0/1074009116 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2243579677 0 0) 0x7f2464142520 con 0x7f24640a98f0 2024-03-25T19:38:21.385 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.385+0000 7f24637fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:21.385 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.385+0000 7f24637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:21.385 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.385+0000 7f24637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:21.385 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.385+0000 7f24637fe640 10 monclient(hunting): my global_id is 4365 2024-03-25T19:38:21.385 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.385+0000 7f24637fe640 1 -- 172.21.15.44:0/1074009116 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24640b2a30 con 0x7f24640a98f0 2024-03-25T19:38:21.387 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.387+0000 7f24637fe640 1 -- 172.21.15.44:0/1074009116 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1557749778 0 0) 0x7f2450003520 con 0x7f246413f710 2024-03-25T19:38:21.387 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.387+0000 7f24637fe640 1 -- 172.21.15.44:0/1074009116 >> v1:172.21.15.111:6789/0 conn(0x7f24641430c0 legacy=0x7f246414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:38:21.387 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.387+0000 7f24637fe640 1 -- 172.21.15.44:0/1074009116 >> v1:172.21.15.44:6789/0 conn(0x7f24640a98f0 legacy=0x7f24640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:21.387 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.387+0000 7f24637fe640 10 monclient: _finish_hunting 0 2024-03-25T19:38:21.387 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.387+0000 7f24637fe640 1 monclient: found mon.noname-b 2024-03-25T19:38:21.387 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.387+0000 7f24637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:21.387 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.387+0000 7f24637fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:38:21.387 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.387+0000 7f24637fe640 1 -- 172.21.15.44:0/1074009116 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24640b44c0 con 0x7f246413f710 2024-03-25T19:38:21.387 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.387+0000 7f24637fe640 10 monclient: _finish_auth 0 2024-03-25T19:38:21.388 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.387+0000 7f24637fe640 10 monclient: _check_auth_tickets 2024-03-25T19:38:21.388 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.387+0000 7f24637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:21.388 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.387+0000 7f24637fe640 1 -- 172.21.15.44:0/1074009116 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f244c001690 con 0x7f246413f710 2024-03-25T19:38:21.388 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.387+0000 7f246ad3a640 5 monclient: authenticate success, global_id 4337 2024-03-25T19:38:21.388 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.387+0000 7f246ad3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:21.388 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.387+0000 7f24637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:21.388 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.387+0000 7f24637fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:21.388 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.387+0000 7f24637fe640 10 monclient: dump: 2024-03-25T19:38:21.388 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:21.388 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:21.388 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:21.388 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:21.388 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:21.388 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:21.388 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:21.389 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:21.389 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:21.389 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:21.389 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.387+0000 7f246ad3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:21.412 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.412+0000 7f24637fe640 1 -- 172.21.15.44:0/1074009116 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f244c0017f0 con 0x7f246413f710 2024-03-25T19:38:21.412 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.412+0000 7f24637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:21.412 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.412+0000 7f24637fe640 1 -- 172.21.15.44:0/1074009116 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f244c001c00 con 0x7f246413f710 2024-03-25T19:38:21.412 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.412+0000 7f24637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:21.412 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.412+0000 7f24637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:21.412 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.412+0000 7f24637fe640 10 monclient: dump: 2024-03-25T19:38:21.413 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:21.413 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:21.413 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:21.413 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:21.413 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:21.413 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:21.413 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:21.413 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:21.413 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:21.413 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:21.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.412+0000 7f246ad3a640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:21.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.412+0000 7f246ad3a640 4 set_mon_vals no callback set 2024-03-25T19:38:21.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.413+0000 7f246ad3a640 10 monclient: shutdown 2024-03-25T19:38:21.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.413+0000 7f246ad3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:21.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.413+0000 7f246ad3a640 1 -- 172.21.15.44:0/1074009116 >> v1:172.21.15.44:6790/0 conn(0x7f246413f710 legacy=0x7f24641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:21.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.413+0000 7f246ad3a640 1 -- 172.21.15.44:0/1074009116 shutdown_connections 2024-03-25T19:38:21.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.413+0000 7f246ad3a640 1 -- 172.21.15.44:0/1074009116 >> 172.21.15.44:0/1074009116 conn(0x7f24640b7ea0 msgr2=0x7f24640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:21.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.413+0000 7f246ad3a640 1 -- 172.21.15.44:0/1074009116 shutdown_connections 2024-03-25T19:38:21.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.413+0000 7f246ad3a640 1 -- 172.21.15.44:0/1074009116 wait complete. 2024-03-25T19:38:21.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.414+0000 7f246ad3a640 10 monclient: build_initial_monmap 2024-03-25T19:38:21.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.414+0000 7f246ad3a640 10 monclient: monmap: 2024-03-25T19:38:21.415 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:38:21.415 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:21.415 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:38:21.415 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:38:21.415 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:21.415 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:38:21.416 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:38:21.416 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:38:21.416 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:38:21.416 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:21.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.414+0000 7f246ad3a640 1 librados: starting msgr at 2024-03-25T19:38:21.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.414+0000 7f246ad3a640 1 librados: starting objecter 2024-03-25T19:38:21.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.414+0000 7f246ad3a640 1 Processor -- start 2024-03-25T19:38:21.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.415+0000 7f246ad3a640 1 -- start start 2024-03-25T19:38:21.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.415+0000 7f246ad3a640 1 librados: setting wanted keys 2024-03-25T19:38:21.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.415+0000 7f246ad3a640 1 librados: calling monclient init 2024-03-25T19:38:21.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.415+0000 7f246ad3a640 10 monclient: init 2024-03-25T19:38:21.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.415+0000 7f246ad3a640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:21.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.415+0000 7f246ad3a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:38:21.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.415+0000 7f246ad3a640 10 monclient(hunting): picked mon.noname-c con 0x7f24640a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T19:38:21.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.415+0000 7f246ad3a640 10 monclient(hunting): picked mon.noname-a con 0x7f246413f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:38:21.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.415+0000 7f246ad3a640 10 monclient(hunting): picked mon.noname-b con 0x7f24641430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:38:21.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.415+0000 7f246ad3a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24641373f0 con 0x7f246413f710 2024-03-25T19:38:21.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.415+0000 7f246ad3a640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24641db280 con 0x7f24641430c0 2024-03-25T19:38:21.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.416+0000 7f246ad3a640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2464138f40 con 0x7f24640a98f0 2024-03-25T19:38:21.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.416+0000 7f246ad3a640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:21.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.416+0000 7f246ad3a640 10 monclient(hunting): authenticate will time out at 5565.586852s 2024-03-25T19:38:21.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.427+0000 7f2468ad6640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f24640a98f0 0x7f24641dab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:48392/0 (socket says 172.21.15.44:48392) 2024-03-25T19:38:21.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.427+0000 7f2468ad6640 1 -- 172.21.15.44:0/2819031055 learned_addr learned my addr 172.21.15.44:0/2819031055 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:38:21.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.428+0000 7f24617fa640 10 client.?.objecter ms_handle_connect 0x7f24640a98f0 2024-03-25T19:38:21.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.428+0000 7f24617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:21.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.441+0000 7f24617fa640 10 client.?.objecter ms_handle_connect 0x7f246413f710 2024-03-25T19:38:21.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.441+0000 7f24617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:21.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.441+0000 7f24617fa640 1 -- 172.21.15.44:0/2819031055 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2402115415 0 0) 0x7f2464138f40 con 0x7f24640a98f0 2024-03-25T19:38:21.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.442+0000 7f24617fa640 10 monclient(hunting): _negotiate 2024-03-25T19:38:21.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.442+0000 7f24617fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:21.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.442+0000 7f24617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:21.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.442+0000 7f24617fa640 10 monclient(hunting): my global_id is 4340 2024-03-25T19:38:21.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.442+0000 7f24617fa640 1 -- 172.21.15.44:0/2819031055 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2444004eb0 con 0x7f24640a98f0 2024-03-25T19:38:21.454 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.454+0000 7f24617fa640 1 -- 172.21.15.44:0/2819031055 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4110117675 0 0) 0x7f2444004eb0 con 0x7f24640a98f0 2024-03-25T19:38:21.454 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.454+0000 7f24617fa640 1 -- 172.21.15.44:0/2819031055 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2464138f40 con 0x7f24640a98f0 2024-03-25T19:38:21.454 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.454+0000 7f24617fa640 1 -- 172.21.15.44:0/2819031055 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f2454001a90 con 0x7f24640a98f0 2024-03-25T19:38:21.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.454+0000 7f24617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:21.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.454+0000 7f24617fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:21.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.454+0000 7f24617fa640 10 monclient(hunting): dump: 2024-03-25T19:38:21.455 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:21.455 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:21.455 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:21.455 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:21.455 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:21.455 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:21.455 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:21.455 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:21.455 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:21.455 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:21.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.454+0000 7f24617fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:21.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.466+0000 7f24617fa640 1 -- 172.21.15.44:0/2819031055 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929854998 0 0) 0x7f24641373f0 con 0x7f246413f710 2024-03-25T19:38:21.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.466+0000 7f24617fa640 10 monclient(hunting): _negotiate 2024-03-25T19:38:21.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.466+0000 7f24617fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:21.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.466+0000 7f24617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:21.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.466+0000 7f24617fa640 10 monclient(hunting): my global_id is 4368 2024-03-25T19:38:21.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.466+0000 7f24617fa640 1 -- 172.21.15.44:0/2819031055 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2444004eb0 con 0x7f246413f710 2024-03-25T19:38:21.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.479+0000 7f24617fa640 1 -- 172.21.15.44:0/2819031055 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3600861914 0 0) 0x7f2464138f40 con 0x7f24640a98f0 2024-03-25T19:38:21.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.479+0000 7f24617fa640 1 -- 172.21.15.44:0/2819031055 >> v1:172.21.15.111:6789/0 conn(0x7f24641430c0 legacy=0x7f24641332c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:21.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.479+0000 7f24617fa640 1 -- 172.21.15.44:0/2819031055 >> v1:172.21.15.44:6789/0 conn(0x7f246413f710 legacy=0x7f2464132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:21.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.479+0000 7f24617fa640 10 monclient: _finish_hunting 0 2024-03-25T19:38:21.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.479+0000 7f24617fa640 1 monclient: found mon.c 2024-03-25T19:38:21.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.479+0000 7f24617fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:21.479 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.479+0000 7f24617fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:38:21.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.479+0000 7f24617fa640 1 -- 172.21.15.44:0/2819031055 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24641339d0 con 0x7f24640a98f0 2024-03-25T19:38:21.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.479+0000 7f24617fa640 10 monclient: _finish_auth 0 2024-03-25T19:38:21.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.479+0000 7f24617fa640 10 monclient: _check_auth_tickets 2024-03-25T19:38:21.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.479+0000 7f24617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:21.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.479+0000 7f246ad3a640 5 monclient: authenticate success, global_id 4340 2024-03-25T19:38:21.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.479+0000 7f246ad3a640 10 monclient: _renew_subs 2024-03-25T19:38:21.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.479+0000 7f246ad3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:38:21.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.479+0000 7f246ad3a640 1 -- 172.21.15.44:0/2819031055 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2464137f50 con 0x7f24640a98f0 2024-03-25T19:38:21.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.479+0000 7f246ad3a640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:21.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.479+0000 7f246ad3a640 10 monclient: _renew_subs 2024-03-25T19:38:21.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.479+0000 7f246ad3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:38:21.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.479+0000 7f246ad3a640 1 -- 172.21.15.44:0/2819031055 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f246413c550 con 0x7f24640a98f0 2024-03-25T19:38:21.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.479+0000 7f246ad3a640 1 librados: init done 2024-03-25T19:38:21.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.480+0000 7f246ad3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:21.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.480+0000 7f246ad3a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:21.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.480+0000 7f246ad3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:38:21.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.480+0000 7f246ad3a640 1 -- 172.21.15.44:0/2819031055 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24640b3ef0 con 0x7f24640a98f0 2024-03-25T19:38:21.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.823+0000 7f24617fa640 1 -- 172.21.15.44:0/2819031055 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2454001d50 con 0x7f24640a98f0 2024-03-25T19:38:21.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.823+0000 7f24617fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:21.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.823+0000 7f24617fa640 1 -- 172.21.15.44:0/2819031055 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f2454003a60 con 0x7f24640a98f0 2024-03-25T19:38:21.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.823+0000 7f24617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:21.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.823+0000 7f24617fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:21.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.823+0000 7f24617fa640 10 monclient: dump: 2024-03-25T19:38:21.823 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:21.823 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:21.823 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:21.824 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:21.824 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:21.824 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:21.824 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:21.824 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:21.824 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:21.824 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:21.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.823+0000 7f2462ffd640 4 set_mon_vals no callback set 2024-03-25T19:38:21.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.824+0000 7f24617fa640 1 -- 172.21.15.44:0/2819031055 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f2454003c80 con 0x7f24640a98f0 2024-03-25T19:38:21.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.824+0000 7f2463fff640 10 monclient: get_auth_request con 0x7f244406dc70 auth_method 0 2024-03-25T19:38:21.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.825+0000 7f24617fa640 10 client.4340.objecter ms_handle_connect 0x7f244406dc70 2024-03-25T19:38:21.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.830+0000 7f24617fa640 1 -- 172.21.15.44:0/2819031055 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f24540549d0 con 0x7f24640a98f0 2024-03-25T19:38:21.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.830+0000 7f24617fa640 10 client.4340.objecter ms_dispatch 0x7f2464000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:21.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.830+0000 7f24617fa640 3 client.4340.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:21.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.830+0000 7f24617fa640 3 client.4340.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:21.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.830+0000 7f24617fa640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-25T19:38:21.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.830+0000 7f24617fa640 1 -- 172.21.15.44:0/2819031055 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2454050310 con 0x7f24640a98f0 2024-03-25T19:38:21.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.830+0000 7f24617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:21.831 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:21.830+0000 7f24617fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:22.055 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:22.054+0000 7f246ad3a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:22.055 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:22.054+0000 7f246ad3a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:22.055 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:22.054+0000 7f246ad3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:38:22.055 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:22.054+0000 7f246ad3a640 1 -- 172.21.15.44:0/2819031055 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24640627b0 con 0x7f24640a98f0 2024-03-25T19:38:22.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:22.101+0000 7f24617fa640 1 -- 172.21.15.44:0/2819031055 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f2454052410 con 0x7f24640a98f0 2024-03-25T19:38:22.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:22.101+0000 7f24617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:22.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:22.101+0000 7f24617fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:22.101 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:38:22.101 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:22.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:22.103+0000 7f246ad3a640 10 librados: watch_flush enter 2024-03-25T19:38:22.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:22.103+0000 7f246ad3a640 10 librados: watch_flush exit 2024-03-25T19:38:22.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:22.103+0000 7f246ad3a640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-25T19:38:22.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:22.103+0000 7f246ad3a640 10 client.4340.objecter successfully canceled tick 2024-03-25T19:38:22.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:22.104+0000 7f246ad3a640 1 -- 172.21.15.44:0/2819031055 >> v1:172.21.15.111:6800/3883215793 conn(0x7f244406dc70 legacy=0x7f2444070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:22.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:22.104+0000 7f246ad3a640 10 monclient: shutdown 2024-03-25T19:38:22.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:22.104+0000 7f246ad3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:22.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:22.104+0000 7f246ad3a640 1 -- 172.21.15.44:0/2819031055 >> v1:172.21.15.44:6790/0 conn(0x7f24640a98f0 legacy=0x7f24641dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:22.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:22.104+0000 7f246ad3a640 1 -- 172.21.15.44:0/2819031055 shutdown_connections 2024-03-25T19:38:22.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:22.104+0000 7f246ad3a640 1 -- 172.21.15.44:0/2819031055 >> 172.21.15.44:0/2819031055 conn(0x7f24640b7ea0 msgr2=0x7f2464074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:22.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:22.104+0000 7f246ad3a640 1 -- 172.21.15.44:0/2819031055 shutdown_connections 2024-03-25T19:38:22.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:22.104+0000 7f246ad3a640 1 -- 172.21.15.44:0/2819031055 wait complete. 2024-03-25T19:38:22.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:22.104+0000 7f246ad3a640 1 librados: shutdown 2024-03-25T19:38:22.117 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:28.923 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:29.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.041+0000 7f33f5ce4640 10 monclient: get_monmap_and_config 2024-03-25T19:38:29.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.041+0000 7f33f5ce4640 10 monclient: build_initial_monmap 2024-03-25T19:38:29.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.041+0000 7f33f5ce4640 10 monclient: monmap: 2024-03-25T19:38:29.041 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:38:29.041 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:29.041 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:38:29.042707+0000 2024-03-25T19:38:29.042 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:38:29.042707+0000 2024-03-25T19:38:29.042 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:29.042 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:29.042 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:38:29.042 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:38:29.042 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:38:29.042 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:29.043 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.042+0000 7f33f5ce4640 1 Processor -- start 2024-03-25T19:38:29.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.042+0000 7f33f5ce4640 1 -- start start 2024-03-25T19:38:29.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.042+0000 7f33f5ce4640 10 monclient: init 2024-03-25T19:38:29.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.043+0000 7f33f5ce4640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:29.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.043+0000 7f33f5ce4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:38:29.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.043+0000 7f33f5ce4640 10 monclient(hunting): picked mon.noname-c con 0x7f33f00a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:38:29.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.043+0000 7f33f5ce4640 10 monclient(hunting): picked mon.noname-a con 0x7f33f013f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:38:29.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.043+0000 7f33f5ce4640 10 monclient(hunting): picked mon.noname-b con 0x7f33f01430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:38:29.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.043+0000 7f33f5ce4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f0142520 con 0x7f33f013f710 2024-03-25T19:38:29.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.043+0000 7f33f5ce4640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f00b28a0 con 0x7f33f00a98f0 2024-03-25T19:38:29.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.043+0000 7f33f5ce4640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f00b2a30 con 0x7f33f01430c0 2024-03-25T19:38:29.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.043+0000 7f33f5ce4640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:29.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.043+0000 7f33f5ce4640 10 monclient(hunting): authenticate will time out at 5573.214274s 2024-03-25T19:38:29.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.050+0000 7f33ef7fe640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f33f00a98f0 0x7f33f00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:60160/0 (socket says 172.21.15.44:60160) 2024-03-25T19:38:29.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.050+0000 7f33ef7fe640 1 -- 172.21.15.44:0/2443945257 learned_addr learned my addr 172.21.15.44:0/2443945257 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:38:29.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.071+0000 7f33ee7fc640 1 -- 172.21.15.44:0/2443945257 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539784901 0 0) 0x7f33f00b28a0 con 0x7f33f00a98f0 2024-03-25T19:38:29.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.071+0000 7f33ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:29.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.071+0000 7f33ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:29.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.071+0000 7f33ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:29.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.071+0000 7f33ee7fc640 10 monclient(hunting): my global_id is 4354 2024-03-25T19:38:29.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.072+0000 7f33ee7fc640 1 -- 172.21.15.44:0/2443945257 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33d8003520 con 0x7f33f00a98f0 2024-03-25T19:38:29.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.074+0000 7f33ee7fc640 1 -- 172.21.15.44:0/2443945257 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3449442465 0 0) 0x7f33f0142520 con 0x7f33f013f710 2024-03-25T19:38:29.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.074+0000 7f33ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:29.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.074+0000 7f33ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:29.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.074+0000 7f33ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:29.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.074+0000 7f33ee7fc640 10 monclient(hunting): my global_id is 4371 2024-03-25T19:38:29.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.074+0000 7f33ee7fc640 1 -- 172.21.15.44:0/2443945257 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33f00b28a0 con 0x7f33f013f710 2024-03-25T19:38:29.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.076+0000 7f33ee7fc640 1 -- 172.21.15.44:0/2443945257 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322266698 0 0) 0x7f33f00b2a30 con 0x7f33f01430c0 2024-03-25T19:38:29.076 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.076+0000 7f33ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:29.076 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.076+0000 7f33ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:29.076 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.076+0000 7f33ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:29.076 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.076+0000 7f33ee7fc640 10 monclient(hunting): my global_id is 4343 2024-03-25T19:38:29.076 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.076+0000 7f33ee7fc640 1 -- 172.21.15.44:0/2443945257 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33f0142520 con 0x7f33f01430c0 2024-03-25T19:38:29.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.084+0000 7f33ee7fc640 1 -- 172.21.15.44:0/2443945257 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2041551715 0 0) 0x7f33d8003520 con 0x7f33f00a98f0 2024-03-25T19:38:29.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.084+0000 7f33ee7fc640 1 -- 172.21.15.44:0/2443945257 >> v1:172.21.15.44:6790/0 conn(0x7f33f01430c0 legacy=0x7f33f014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:29.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.084+0000 7f33ee7fc640 1 -- 172.21.15.44:0/2443945257 >> v1:172.21.15.44:6789/0 conn(0x7f33f013f710 legacy=0x7f33f01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:29.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.084+0000 7f33ee7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:38:29.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.084+0000 7f33ee7fc640 1 monclient: found mon.noname-c 2024-03-25T19:38:29.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.084+0000 7f33ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:29.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.084+0000 7f33ee7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:38:29.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.084+0000 7f33ee7fc640 1 -- 172.21.15.44:0/2443945257 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f00b44c0 con 0x7f33f00a98f0 2024-03-25T19:38:29.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.084+0000 7f33ee7fc640 10 monclient: _finish_auth 0 2024-03-25T19:38:29.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.084+0000 7f33ee7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:38:29.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.084+0000 7f33ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:29.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.084+0000 7f33ee7fc640 1 -- 172.21.15.44:0/2443945257 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f33e0001690 con 0x7f33f00a98f0 2024-03-25T19:38:29.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.084+0000 7f33f5ce4640 5 monclient: authenticate success, global_id 4354 2024-03-25T19:38:29.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.084+0000 7f33f5ce4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:29.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.084+0000 7f33ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:29.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.084+0000 7f33ee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:29.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.084+0000 7f33ee7fc640 10 monclient: dump: 2024-03-25T19:38:29.085 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:29.085 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:29.085 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:29.085 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:29.086 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:29.086 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:29.086 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:29.086 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:29.086 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:29.086 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:29.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.084+0000 7f33f5ce4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:29.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.106+0000 7f33ee7fc640 1 -- 172.21.15.44:0/2443945257 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33e00017f0 con 0x7f33f00a98f0 2024-03-25T19:38:29.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.106+0000 7f33ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:29.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.106+0000 7f33ee7fc640 1 -- 172.21.15.44:0/2443945257 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f33e0001c00 con 0x7f33f00a98f0 2024-03-25T19:38:29.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.106+0000 7f33f5ce4640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:29.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.106+0000 7f33f5ce4640 4 set_mon_vals no callback set 2024-03-25T19:38:29.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.106+0000 7f33ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:29.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.106+0000 7f33ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:29.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.106+0000 7f33ee7fc640 10 monclient: dump: 2024-03-25T19:38:29.106 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:29.107 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:29.107 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:29.107 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:29.107 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:29.107 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:29.107 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:29.107 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:29.107 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:29.107 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:29.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.107+0000 7f33f5ce4640 10 monclient: shutdown 2024-03-25T19:38:29.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.107+0000 7f33f5ce4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:29.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.107+0000 7f33f5ce4640 1 -- 172.21.15.44:0/2443945257 >> v1:172.21.15.111:6789/0 conn(0x7f33f00a98f0 legacy=0x7f33f00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:29.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.107+0000 7f33f5ce4640 1 -- 172.21.15.44:0/2443945257 shutdown_connections 2024-03-25T19:38:29.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.107+0000 7f33f5ce4640 1 -- 172.21.15.44:0/2443945257 >> 172.21.15.44:0/2443945257 conn(0x7f33f00b7ea0 msgr2=0x7f33f00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:29.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.107+0000 7f33f5ce4640 1 -- 172.21.15.44:0/2443945257 shutdown_connections 2024-03-25T19:38:29.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.107+0000 7f33f5ce4640 1 -- 172.21.15.44:0/2443945257 wait complete. 2024-03-25T19:38:29.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.108+0000 7f33f5ce4640 10 monclient: build_initial_monmap 2024-03-25T19:38:29.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.108+0000 7f33f5ce4640 10 monclient: monmap: 2024-03-25T19:38:29.109 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:38:29.110 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:29.110 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:38:29.110 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:38:29.110 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:29.110 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:38:29.110 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:38:29.110 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:38:29.110 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:38:29.110 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:29.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.108+0000 7f33f5ce4640 1 librados: starting msgr at 2024-03-25T19:38:29.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.108+0000 7f33f5ce4640 1 librados: starting objecter 2024-03-25T19:38:29.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.108+0000 7f33f5ce4640 1 Processor -- start 2024-03-25T19:38:29.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.109+0000 7f33f5ce4640 1 -- start start 2024-03-25T19:38:29.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.109+0000 7f33f5ce4640 1 librados: setting wanted keys 2024-03-25T19:38:29.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.109+0000 7f33f5ce4640 1 librados: calling monclient init 2024-03-25T19:38:29.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.109+0000 7f33f5ce4640 10 monclient: init 2024-03-25T19:38:29.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.110+0000 7f33f5ce4640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:29.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.110+0000 7f33f5ce4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:38:29.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.110+0000 7f33f5ce4640 10 monclient(hunting): picked mon.noname-b con 0x7f33f00a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:38:29.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.110+0000 7f33f5ce4640 10 monclient(hunting): picked mon.noname-c con 0x7f33f013f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:38:29.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.110+0000 7f33f5ce4640 10 monclient(hunting): picked mon.noname-a con 0x7f33f01430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:38:29.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.110+0000 7f33f5ce4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f0135590 con 0x7f33f01430c0 2024-03-25T19:38:29.112 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.110+0000 7f33f5ce4640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f0136e50 con 0x7f33f00a98f0 2024-03-25T19:38:29.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.110+0000 7f33f5ce4640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f0135070 con 0x7f33f013f710 2024-03-25T19:38:29.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.110+0000 7f33f5ce4640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:29.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.110+0000 7f33f5ce4640 10 monclient(hunting): authenticate will time out at 5573.281753s 2024-03-25T19:38:29.120 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.120+0000 7f33eeffd640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f33f013f710 0x7f33f01d7750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:38544/0 (socket says 172.21.15.44:38544) 2024-03-25T19:38:29.120 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.120+0000 7f33eeffd640 1 -- 172.21.15.44:0/18099975 learned_addr learned my addr 172.21.15.44:0/18099975 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:38:29.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.122+0000 7f33d7fff640 10 client.?.objecter ms_handle_connect 0x7f33f013f710 2024-03-25T19:38:29.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.122+0000 7f33d7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:29.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.123+0000 7f33d7fff640 10 client.?.objecter ms_handle_connect 0x7f33f00a98f0 2024-03-25T19:38:29.123 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.123+0000 7f33d7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:29.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.128+0000 7f33d7fff640 1 -- 172.21.15.44:0/18099975 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702107670 0 0) 0x7f33f0135070 con 0x7f33f013f710 2024-03-25T19:38:29.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.128+0000 7f33d7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:29.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.128+0000 7f33d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:29.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.128+0000 7f33d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:29.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.128+0000 7f33d7fff640 10 monclient(hunting): my global_id is 4346 2024-03-25T19:38:29.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.128+0000 7f33d7fff640 1 -- 172.21.15.44:0/18099975 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33dc004eb0 con 0x7f33f013f710 2024-03-25T19:38:29.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.130+0000 7f33d7fff640 1 -- 172.21.15.44:0/18099975 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1987157028 0 0) 0x7f33f0136e50 con 0x7f33f00a98f0 2024-03-25T19:38:29.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.130+0000 7f33d7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:29.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.130+0000 7f33d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:29.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.130+0000 7f33d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:29.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.130+0000 7f33d7fff640 10 monclient(hunting): my global_id is 4357 2024-03-25T19:38:29.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.131+0000 7f33d7fff640 1 -- 172.21.15.44:0/18099975 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33f0135070 con 0x7f33f00a98f0 2024-03-25T19:38:29.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.140+0000 7f33d7fff640 10 client.?.objecter ms_handle_connect 0x7f33f01430c0 2024-03-25T19:38:29.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.140+0000 7f33d7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:29.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.141+0000 7f33d7fff640 1 -- 172.21.15.44:0/18099975 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3838001046 0 0) 0x7f33f0135070 con 0x7f33f00a98f0 2024-03-25T19:38:29.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.141+0000 7f33d7fff640 1 -- 172.21.15.44:0/18099975 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33f0136e50 con 0x7f33f00a98f0 2024-03-25T19:38:29.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.144+0000 7f33d7fff640 1 -- 172.21.15.44:0/18099975 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f33e0001b40 con 0x7f33f00a98f0 2024-03-25T19:38:29.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.144+0000 7f33d7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:29.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.144+0000 7f33d7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:29.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.144+0000 7f33d7fff640 10 monclient(hunting): dump: 2024-03-25T19:38:29.144 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:29.144 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:29.144 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:29.144 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:29.144 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:29.144 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:29.144 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:29.144 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:29.145 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:29.145 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:29.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.144+0000 7f33d7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:29.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.146+0000 7f33d7fff640 1 -- 172.21.15.44:0/18099975 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 339559648 0 0) 0x7f33dc004eb0 con 0x7f33f013f710 2024-03-25T19:38:29.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.146+0000 7f33d7fff640 1 -- 172.21.15.44:0/18099975 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33f0135070 con 0x7f33f013f710 2024-03-25T19:38:29.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.146+0000 7f33d7fff640 1 -- 172.21.15.44:0/18099975 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f33e4001a30 con 0x7f33f013f710 2024-03-25T19:38:29.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.146+0000 7f33d7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:29.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.146+0000 7f33d7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:29.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.146+0000 7f33d7fff640 10 monclient(hunting): dump: 2024-03-25T19:38:29.147 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:29.147 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:29.147 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:29.147 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:29.147 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:29.147 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:29.147 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:29.148 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:29.148 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:29.148 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:29.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.148+0000 7f33d7fff640 1 -- 172.21.15.44:0/18099975 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4080169896 0 0) 0x7f33f0135590 con 0x7f33f01430c0 2024-03-25T19:38:29.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.148+0000 7f33d7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:29.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.148+0000 7f33d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:29.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.148+0000 7f33d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:29.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.148+0000 7f33d7fff640 10 monclient(hunting): my global_id is 4374 2024-03-25T19:38:29.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.148+0000 7f33d7fff640 1 -- 172.21.15.44:0/18099975 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33dc004eb0 con 0x7f33f01430c0 2024-03-25T19:38:29.154 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.154+0000 7f33d7fff640 1 -- 172.21.15.44:0/18099975 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 974626335 0 0) 0x7f33f0136e50 con 0x7f33f00a98f0 2024-03-25T19:38:29.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.154+0000 7f33d7fff640 1 -- 172.21.15.44:0/18099975 >> v1:172.21.15.44:6790/0 conn(0x7f33f013f710 legacy=0x7f33f01d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:29.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.154+0000 7f33d7fff640 1 -- 172.21.15.44:0/18099975 >> v1:172.21.15.44:6789/0 conn(0x7f33f01430c0 legacy=0x7f33f01e3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:29.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.154+0000 7f33d7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:38:29.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.154+0000 7f33d7fff640 1 monclient: found mon.b 2024-03-25T19:38:29.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.154+0000 7f33d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:29.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.154+0000 7f33d7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:29.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.154+0000 7f33d7fff640 1 -- 172.21.15.44:0/18099975 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f0135d80 con 0x7f33f00a98f0 2024-03-25T19:38:29.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.154+0000 7f33d7fff640 10 monclient: _finish_auth 0 2024-03-25T19:38:29.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.154+0000 7f33d7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:38:29.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.154+0000 7f33d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:29.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.154+0000 7f33f5ce4640 5 monclient: authenticate success, global_id 4357 2024-03-25T19:38:29.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.154+0000 7f33f5ce4640 10 monclient: _renew_subs 2024-03-25T19:38:29.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.154+0000 7f33f5ce4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:29.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.154+0000 7f33f5ce4640 1 -- 172.21.15.44:0/18099975 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f0131660 con 0x7f33f00a98f0 2024-03-25T19:38:29.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.154+0000 7f33f5ce4640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:29.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.154+0000 7f33f5ce4640 10 monclient: _renew_subs 2024-03-25T19:38:29.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.154+0000 7f33f5ce4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:29.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.154+0000 7f33f5ce4640 1 -- 172.21.15.44:0/18099975 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f00b78a0 con 0x7f33f00a98f0 2024-03-25T19:38:29.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.154+0000 7f33f5ce4640 1 librados: init done 2024-03-25T19:38:29.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.155+0000 7f33f5ce4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:29.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.155+0000 7f33f5ce4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:29.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.155+0000 7f33f5ce4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:29.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.155+0000 7f33f5ce4640 1 -- 172.21.15.44:0/18099975 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33f0135f70 con 0x7f33f00a98f0 2024-03-25T19:38:29.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.432+0000 7f33d7fff640 1 -- 172.21.15.44:0/18099975 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33e0001ca0 con 0x7f33f00a98f0 2024-03-25T19:38:29.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.432+0000 7f33d7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:29.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.432+0000 7f33d7fff640 1 -- 172.21.15.44:0/18099975 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f33e0004a90 con 0x7f33f00a98f0 2024-03-25T19:38:29.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.432+0000 7f33d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:29.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.432+0000 7f33d7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:29.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.432+0000 7f33d7fff640 10 monclient: dump: 2024-03-25T19:38:29.433 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:29.433 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:29.433 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:29.433 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:29.433 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:29.433 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:29.433 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:29.433 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:29.433 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:29.433 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:29.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.432+0000 7f33edffb640 4 set_mon_vals no callback set 2024-03-25T19:38:29.434 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.433+0000 7f33d7fff640 1 -- 172.21.15.44:0/18099975 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f33e001aa60 con 0x7f33f00a98f0 2024-03-25T19:38:29.434 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.434+0000 7f33eeffd640 10 monclient: get_auth_request con 0x7f33dc06df00 auth_method 0 2024-03-25T19:38:29.434 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.434+0000 7f33d7fff640 10 client.4357.objecter ms_handle_connect 0x7f33dc06df00 2024-03-25T19:38:29.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.437+0000 7f33d7fff640 1 -- 172.21.15.44:0/18099975 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f33e0054740 con 0x7f33f00a98f0 2024-03-25T19:38:29.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.437+0000 7f33d7fff640 10 client.4357.objecter ms_dispatch 0x7f33f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:29.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.437+0000 7f33d7fff640 3 client.4357.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:29.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.437+0000 7f33d7fff640 3 client.4357.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:29.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.437+0000 7f33d7fff640 20 client.4357.objecter dump_active .. 0 homeless 2024-03-25T19:38:29.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.438+0000 7f33d7fff640 1 -- 172.21.15.44:0/18099975 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f33e0050080 con 0x7f33f00a98f0 2024-03-25T19:38:29.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.438+0000 7f33d7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:29.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.438+0000 7f33d7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:29.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.691+0000 7f33f5ce4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:29.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.691+0000 7f33f5ce4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:29.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.691+0000 7f33f5ce4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:29.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.691+0000 7f33f5ce4640 1 -- 172.21.15.44:0/18099975 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33f0135200 con 0x7f33f00a98f0 2024-03-25T19:38:29.716 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.716+0000 7f33d7fff640 1 -- 172.21.15.44:0/18099975 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f33e0052180 con 0x7f33f00a98f0 2024-03-25T19:38:29.716 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.716+0000 7f33d7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:29.717 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.716+0000 7f33d7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:29.717 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:38:29.717 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:29.719 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.719+0000 7f33f5ce4640 10 librados: watch_flush enter 2024-03-25T19:38:29.719 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.719+0000 7f33f5ce4640 10 librados: watch_flush exit 2024-03-25T19:38:29.719 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.719+0000 7f33f5ce4640 20 client.4357.objecter shutdown clearing up homeless session... 2024-03-25T19:38:29.719 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.719+0000 7f33f5ce4640 10 client.4357.objecter successfully canceled tick 2024-03-25T19:38:29.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.719+0000 7f33f5ce4640 1 -- 172.21.15.44:0/18099975 >> v1:172.21.15.111:6800/3883215793 conn(0x7f33dc06df00 legacy=0x7f33dc0703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:29.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.719+0000 7f33f5ce4640 10 monclient: shutdown 2024-03-25T19:38:29.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.719+0000 7f33f5ce4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:29.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.719+0000 7f33f5ce4640 1 -- 172.21.15.44:0/18099975 >> v1:172.21.15.111:6789/0 conn(0x7f33f00a98f0 legacy=0x7f33f0130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:29.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.719+0000 7f33f5ce4640 1 -- 172.21.15.44:0/18099975 shutdown_connections 2024-03-25T19:38:29.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.719+0000 7f33f5ce4640 1 -- 172.21.15.44:0/18099975 >> 172.21.15.44:0/18099975 conn(0x7f33f00b7ea0 msgr2=0x7f33f0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:29.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.719+0000 7f33f5ce4640 1 -- 172.21.15.44:0/18099975 shutdown_connections 2024-03-25T19:38:29.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.719+0000 7f33f5ce4640 1 -- 172.21.15.44:0/18099975 wait complete. 2024-03-25T19:38:29.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:29.719+0000 7f33f5ce4640 1 librados: shutdown 2024-03-25T19:38:29.732 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:36.539 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:36.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.640+0000 7f1fcf44a640 10 monclient: get_monmap_and_config 2024-03-25T19:38:36.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.640+0000 7f1fcf44a640 10 monclient: build_initial_monmap 2024-03-25T19:38:36.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.640+0000 7f1fcf44a640 10 monclient: monmap: 2024-03-25T19:38:36.641 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:38:36.641 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:36.641 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:38:36.641844+0000 2024-03-25T19:38:36.641 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:38:36.641844+0000 2024-03-25T19:38:36.641 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:36.641 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:36.641 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:38:36.641 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:38:36.641 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:38:36.641 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:36.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.641+0000 7f1fcf44a640 1 Processor -- start 2024-03-25T19:38:36.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.641+0000 7f1fcf44a640 1 -- start start 2024-03-25T19:38:36.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.641+0000 7f1fcf44a640 10 monclient: init 2024-03-25T19:38:36.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.642+0000 7f1fcf44a640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:36.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.642+0000 7f1fcf44a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:38:36.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.642+0000 7f1fcf44a640 10 monclient(hunting): picked mon.noname-a con 0x7f1fc80a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:38:36.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.642+0000 7f1fcf44a640 10 monclient(hunting): picked mon.noname-b con 0x7f1fc813f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:38:36.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.642+0000 7f1fcf44a640 10 monclient(hunting): picked mon.noname-c con 0x7f1fc81430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:38:36.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.642+0000 7f1fcf44a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc8142520 con 0x7f1fc80a98f0 2024-03-25T19:38:36.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.642+0000 7f1fcf44a640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc80b28a0 con 0x7f1fc81430c0 2024-03-25T19:38:36.644 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.642+0000 7f1fcf44a640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc80b2a30 con 0x7f1fc813f710 2024-03-25T19:38:36.644 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.642+0000 7f1fcf44a640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:36.644 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.642+0000 7f1fcf44a640 10 monclient(hunting): authenticate will time out at 5580.813477s 2024-03-25T19:38:36.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.666+0000 7f1fcd1e6640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f1fc80a98f0 0x7f1fc80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:53194/0 (socket says 172.21.15.44:53194) 2024-03-25T19:38:36.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.666+0000 7f1fcd1e6640 1 -- 172.21.15.44:0/3426930851 learned_addr learned my addr 172.21.15.44:0/3426930851 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:38:36.683 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.683+0000 7f1fbffff640 1 -- 172.21.15.44:0/3426930851 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578054619 0 0) 0x7f1fc8142520 con 0x7f1fc80a98f0 2024-03-25T19:38:36.683 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.683+0000 7f1fbffff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:36.683 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.683+0000 7f1fbffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:36.683 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.683+0000 7f1fbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:36.683 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.683+0000 7f1fbffff640 10 monclient(hunting): my global_id is 4377 2024-03-25T19:38:36.683 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.683+0000 7f1fbffff640 1 -- 172.21.15.44:0/3426930851 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fb4003520 con 0x7f1fc80a98f0 2024-03-25T19:38:36.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.727+0000 7f1fbffff640 1 -- 172.21.15.44:0/3426930851 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 530435455 0 0) 0x7f1fb4003520 con 0x7f1fc80a98f0 2024-03-25T19:38:36.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.727+0000 7f1fbffff640 1 -- 172.21.15.44:0/3426930851 >> v1:172.21.15.44:6790/0 conn(0x7f1fc813f710 legacy=0x7f1fc81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:36.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.727+0000 7f1fbffff640 1 -- 172.21.15.44:0/3426930851 >> v1:172.21.15.111:6789/0 conn(0x7f1fc81430c0 legacy=0x7f1fc814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:36.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.727+0000 7f1fbffff640 10 monclient: _finish_hunting 0 2024-03-25T19:38:36.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.727+0000 7f1fbffff640 1 monclient: found mon.noname-a 2024-03-25T19:38:36.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.727+0000 7f1fbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:36.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.727+0000 7f1fbffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:38:36.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.727+0000 7f1fbffff640 1 -- 172.21.15.44:0/3426930851 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc80b44c0 con 0x7f1fc80a98f0 2024-03-25T19:38:36.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.727+0000 7f1fbffff640 10 monclient: _finish_auth 0 2024-03-25T19:38:36.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.727+0000 7f1fbffff640 10 monclient: _check_auth_tickets 2024-03-25T19:38:36.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.727+0000 7f1fbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:36.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.727+0000 7f1fbffff640 1 -- 172.21.15.44:0/3426930851 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f1fc4001690 con 0x7f1fc80a98f0 2024-03-25T19:38:36.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.727+0000 7f1fcf44a640 5 monclient: authenticate success, global_id 4377 2024-03-25T19:38:36.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.727+0000 7f1fcf44a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:36.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.727+0000 7f1fbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:36.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.727+0000 7f1fbffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:38:36.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.727+0000 7f1fbffff640 10 monclient: dump: 2024-03-25T19:38:36.728 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:36.728 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:36.729 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:36.729 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:36.729 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:36.729 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:36.729 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:36.729 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:36.729 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:36.729 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:36.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.727+0000 7f1fcf44a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:36.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.804+0000 7f1fbffff640 1 -- 172.21.15.44:0/3426930851 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fc40017f0 con 0x7f1fc80a98f0 2024-03-25T19:38:36.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.804+0000 7f1fbffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:36.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.804+0000 7f1fbffff640 1 -- 172.21.15.44:0/3426930851 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f1fc4001c00 con 0x7f1fc80a98f0 2024-03-25T19:38:36.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.804+0000 7f1fbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:36.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.804+0000 7f1fbffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:36.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.804+0000 7f1fbffff640 10 monclient: dump: 2024-03-25T19:38:36.804 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:36.804 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:36.805 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:36.805 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:36.805 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:36.805 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:36.805 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:36.805 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:36.805 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:36.805 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:36.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.804+0000 7f1fcf44a640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:36.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.804+0000 7f1fcf44a640 4 set_mon_vals no callback set 2024-03-25T19:38:36.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.805+0000 7f1fcf44a640 10 monclient: shutdown 2024-03-25T19:38:36.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.805+0000 7f1fcf44a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:36.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.805+0000 7f1fcf44a640 1 -- 172.21.15.44:0/3426930851 >> v1:172.21.15.44:6789/0 conn(0x7f1fc80a98f0 legacy=0x7f1fc80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:36.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.805+0000 7f1fcf44a640 1 -- 172.21.15.44:0/3426930851 shutdown_connections 2024-03-25T19:38:36.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.805+0000 7f1fcf44a640 1 -- 172.21.15.44:0/3426930851 >> 172.21.15.44:0/3426930851 conn(0x7f1fc80b7ea0 msgr2=0x7f1fc80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:36.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.805+0000 7f1fcf44a640 1 -- 172.21.15.44:0/3426930851 shutdown_connections 2024-03-25T19:38:36.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.805+0000 7f1fcf44a640 1 -- 172.21.15.44:0/3426930851 wait complete. 2024-03-25T19:38:36.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.806+0000 7f1fcf44a640 10 monclient: build_initial_monmap 2024-03-25T19:38:36.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.806+0000 7f1fcf44a640 10 monclient: monmap: 2024-03-25T19:38:36.807 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:38:36.807 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:36.807 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:38:36.808 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:38:36.808 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:36.808 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:38:36.808 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:38:36.808 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:38:36.808 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:38:36.808 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:36.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.806+0000 7f1fcf44a640 1 librados: starting msgr at 2024-03-25T19:38:36.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.806+0000 7f1fcf44a640 1 librados: starting objecter 2024-03-25T19:38:36.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.807+0000 7f1fcf44a640 1 Processor -- start 2024-03-25T19:38:36.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.807+0000 7f1fcf44a640 1 -- start start 2024-03-25T19:38:36.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.807+0000 7f1fcf44a640 1 librados: setting wanted keys 2024-03-25T19:38:36.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.807+0000 7f1fcf44a640 1 librados: calling monclient init 2024-03-25T19:38:36.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.807+0000 7f1fcf44a640 10 monclient: init 2024-03-25T19:38:36.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.807+0000 7f1fcf44a640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:36.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.807+0000 7f1fcf44a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:38:36.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.807+0000 7f1fcf44a640 10 monclient(hunting): picked mon.noname-a con 0x7f1fc80a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:38:36.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.807+0000 7f1fcf44a640 10 monclient(hunting): picked mon.noname-c con 0x7f1fc813f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:38:36.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.807+0000 7f1fcf44a640 10 monclient(hunting): picked mon.noname-b con 0x7f1fc81430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:38:36.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.808+0000 7f1fcf44a640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc8138c70 con 0x7f1fc80a98f0 2024-03-25T19:38:36.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.808+0000 7f1fcf44a640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc81367d0 con 0x7f1fc81430c0 2024-03-25T19:38:36.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.808+0000 7f1fcf44a640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc8136960 con 0x7f1fc813f710 2024-03-25T19:38:36.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.808+0000 7f1fcf44a640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:36.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.808+0000 7f1fcf44a640 10 monclient(hunting): authenticate will time out at 5580.978926s 2024-03-25T19:38:36.811 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.812+0000 7f1fcc9e5640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f1fc813f710 0x7f1fc8132f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:43252/0 (socket says 172.21.15.44:43252) 2024-03-25T19:38:36.811 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.812+0000 7f1fcc9e5640 1 -- 172.21.15.44:0/3217565867 learned_addr learned my addr 172.21.15.44:0/3217565867 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:38:36.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.812+0000 7f1fbdffb640 10 client.?.objecter ms_handle_connect 0x7f1fc813f710 2024-03-25T19:38:36.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.812+0000 7f1fbdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:36.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.830+0000 7f1fbdffb640 10 client.?.objecter ms_handle_connect 0x7f1fc81430c0 2024-03-25T19:38:36.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.830+0000 7f1fbdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:36.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.830+0000 7f1fbdffb640 1 -- 172.21.15.44:0/3217565867 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709256008 0 0) 0x7f1fc8136960 con 0x7f1fc813f710 2024-03-25T19:38:36.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.830+0000 7f1fbdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:36.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.830+0000 7f1fbdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:36.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.830+0000 7f1fbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:36.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.830+0000 7f1fbdffb640 10 monclient(hunting): my global_id is 4349 2024-03-25T19:38:36.830 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.830+0000 7f1fbdffb640 1 -- 172.21.15.44:0/3217565867 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fac004eb0 con 0x7f1fc813f710 2024-03-25T19:38:36.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.833+0000 7f1fbdffb640 1 -- 172.21.15.44:0/3217565867 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366788277 0 0) 0x7f1fc81367d0 con 0x7f1fc81430c0 2024-03-25T19:38:36.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.833+0000 7f1fbdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:36.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.833+0000 7f1fbdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:36.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.833+0000 7f1fbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:36.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.833+0000 7f1fbdffb640 10 monclient(hunting): my global_id is 4360 2024-03-25T19:38:36.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.833+0000 7f1fbdffb640 1 -- 172.21.15.44:0/3217565867 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fc8136960 con 0x7f1fc81430c0 2024-03-25T19:38:36.834 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.835+0000 7f1fbdffb640 10 client.?.objecter ms_handle_connect 0x7f1fc80a98f0 2024-03-25T19:38:36.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.835+0000 7f1fbdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:36.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.842+0000 7f1fbdffb640 1 -- 172.21.15.44:0/3217565867 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284849890 0 0) 0x7f1fc8138c70 con 0x7f1fc80a98f0 2024-03-25T19:38:36.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.842+0000 7f1fbdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:36.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.842+0000 7f1fbdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:36.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.842+0000 7f1fbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:36.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.842+0000 7f1fbdffb640 10 monclient(hunting): my global_id is 4380 2024-03-25T19:38:36.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.842+0000 7f1fbdffb640 1 -- 172.21.15.44:0/3217565867 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fc81367d0 con 0x7f1fc80a98f0 2024-03-25T19:38:36.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.843+0000 7f1fbdffb640 1 -- 172.21.15.44:0/3217565867 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1099204735 0 0) 0x7f1fc8136960 con 0x7f1fc81430c0 2024-03-25T19:38:36.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.843+0000 7f1fbdffb640 1 -- 172.21.15.44:0/3217565867 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fc8138c70 con 0x7f1fc81430c0 2024-03-25T19:38:36.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.843+0000 7f1fbdffb640 1 -- 172.21.15.44:0/3217565867 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f1fc0001b90 con 0x7f1fc81430c0 2024-03-25T19:38:36.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.844+0000 7f1fbdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:36.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.844+0000 7f1fbdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:36.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.844+0000 7f1fbdffb640 10 monclient(hunting): dump: 2024-03-25T19:38:36.844 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:36.844 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:36.844 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:36.844 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:36.844 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:36.844 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:36.844 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:36.844 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:36.845 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:36.845 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:36.845 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.844+0000 7f1fbdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:36.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.847+0000 7f1fbdffb640 1 -- 172.21.15.44:0/3217565867 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 379874790 0 0) 0x7f1fac004eb0 con 0x7f1fc813f710 2024-03-25T19:38:36.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.847+0000 7f1fbdffb640 1 -- 172.21.15.44:0/3217565867 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fc8136960 con 0x7f1fc813f710 2024-03-25T19:38:36.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.848+0000 7f1fbdffb640 1 -- 172.21.15.44:0/3217565867 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f1fb8001ad0 con 0x7f1fc813f710 2024-03-25T19:38:36.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.848+0000 7f1fbdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:36.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.848+0000 7f1fbdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:36.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.848+0000 7f1fbdffb640 10 monclient(hunting): dump: 2024-03-25T19:38:36.848 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:36.848 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:36.848 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:36.848 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:36.848 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:36.848 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:36.848 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:36.848 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:36.849 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:36.849 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:36.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.854+0000 7f1fbdffb640 1 -- 172.21.15.44:0/3217565867 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2834800201 0 0) 0x7f1fc8138c70 con 0x7f1fc81430c0 2024-03-25T19:38:36.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.854+0000 7f1fbdffb640 1 -- 172.21.15.44:0/3217565867 >> v1:172.21.15.44:6790/0 conn(0x7f1fc813f710 legacy=0x7f1fc8132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:36.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.854+0000 7f1fbdffb640 1 -- 172.21.15.44:0/3217565867 >> v1:172.21.15.44:6789/0 conn(0x7f1fc80a98f0 legacy=0x7f1fc8137610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:36.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.854+0000 7f1fbdffb640 10 monclient: _finish_hunting 0 2024-03-25T19:38:36.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.854+0000 7f1fbdffb640 1 monclient: found mon.b 2024-03-25T19:38:36.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.854+0000 7f1fbdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:36.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.854+0000 7f1fbdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:36.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.854+0000 7f1fbdffb640 1 -- 172.21.15.44:0/3217565867 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc8139630 con 0x7f1fc81430c0 2024-03-25T19:38:36.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.854+0000 7f1fbdffb640 10 monclient: _finish_auth 0 2024-03-25T19:38:36.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.854+0000 7f1fbdffb640 10 monclient: _check_auth_tickets 2024-03-25T19:38:36.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.854+0000 7f1fbdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:36.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.854+0000 7f1fcf44a640 5 monclient: authenticate success, global_id 4360 2024-03-25T19:38:36.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.854+0000 7f1fcf44a640 10 monclient: _renew_subs 2024-03-25T19:38:36.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.854+0000 7f1fcf44a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:36.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.854+0000 7f1fcf44a640 1 -- 172.21.15.44:0/3217565867 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc8139880 con 0x7f1fc81430c0 2024-03-25T19:38:36.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.854+0000 7f1fcf44a640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:36.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.854+0000 7f1fcf44a640 10 monclient: _renew_subs 2024-03-25T19:38:36.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.854+0000 7f1fcf44a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:36.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.854+0000 7f1fcf44a640 1 -- 172.21.15.44:0/3217565867 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc81337f0 con 0x7f1fc81430c0 2024-03-25T19:38:36.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.854+0000 7f1fcf44a640 1 librados: init done 2024-03-25T19:38:36.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.855+0000 7f1fcf44a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:36.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.855+0000 7f1fcf44a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:36.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.855+0000 7f1fcf44a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:36.857 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.855+0000 7f1fcf44a640 1 -- 172.21.15.44:0/3217565867 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc80627b0 con 0x7f1fc81430c0 2024-03-25T19:38:36.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.874+0000 7f1fbdffb640 1 -- 172.21.15.44:0/3217565867 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fc0002040 con 0x7f1fc81430c0 2024-03-25T19:38:36.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.874+0000 7f1fbdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:36.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.874+0000 7f1fbdffb640 1 -- 172.21.15.44:0/3217565867 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f1fc0003b30 con 0x7f1fc81430c0 2024-03-25T19:38:36.875 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.874+0000 7f1fbdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:36.875 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.874+0000 7f1fbdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:36.875 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.874+0000 7f1fbdffb640 10 monclient: dump: 2024-03-25T19:38:36.875 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:36.875 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:36.875 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:36.875 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:36.875 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:36.875 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:36.875 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:36.875 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:36.875 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:36.875 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:36.876 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.874+0000 7f1fbf7fe640 4 set_mon_vals no callback set 2024-03-25T19:38:36.876 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.875+0000 7f1fbdffb640 1 -- 172.21.15.44:0/3217565867 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f1fc0003d50 con 0x7f1fc81430c0 2024-03-25T19:38:36.876 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.876+0000 7f1fcd1e6640 10 monclient: get_auth_request con 0x7f1fac06def0 auth_method 0 2024-03-25T19:38:36.876 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.876+0000 7f1fbdffb640 10 client.4360.objecter ms_handle_connect 0x7f1fac06def0 2024-03-25T19:38:36.877 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.877+0000 7f1fbdffb640 1 -- 172.21.15.44:0/3217565867 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f1fc0053850 con 0x7f1fc81430c0 2024-03-25T19:38:36.877 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.877+0000 7f1fbdffb640 10 client.4360.objecter ms_dispatch 0x7f1fc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:36.877 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.877+0000 7f1fbdffb640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:36.877 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.877+0000 7f1fbdffb640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:36.877 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:36.877+0000 7f1fbdffb640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-25T19:38:37.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:37.149+0000 7f1fbdffb640 1 -- 172.21.15.44:0/3217565867 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1fc004f360 con 0x7f1fc81430c0 2024-03-25T19:38:37.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:37.149+0000 7f1fbdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:37.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:37.149+0000 7f1fbdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:37.402 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:37.402+0000 7f1fcf44a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:37.402 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:37.402+0000 7f1fcf44a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:37.402 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:37.402+0000 7f1fcf44a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:37.402 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:37.402+0000 7f1fcf44a640 1 -- 172.21.15.44:0/3217565867 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fc8074cc0 con 0x7f1fc81430c0 2024-03-25T19:38:37.434 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:37.434+0000 7f1fbdffb640 1 -- 172.21.15.44:0/3217565867 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f1fc0051290 con 0x7f1fc81430c0 2024-03-25T19:38:37.434 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:37.434+0000 7f1fbdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:37.434 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:37.434+0000 7f1fbdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:37.435 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:38:37.435 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:37.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:37.437+0000 7f1fcf44a640 10 librados: watch_flush enter 2024-03-25T19:38:37.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:37.437+0000 7f1fcf44a640 10 librados: watch_flush exit 2024-03-25T19:38:37.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:37.437+0000 7f1fcf44a640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-25T19:38:37.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:37.437+0000 7f1fcf44a640 10 client.4360.objecter successfully canceled tick 2024-03-25T19:38:37.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:37.437+0000 7f1fcf44a640 1 -- 172.21.15.44:0/3217565867 >> v1:172.21.15.111:6800/3883215793 conn(0x7f1fac06def0 legacy=0x7f1fac0703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:37.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:37.437+0000 7f1fcf44a640 10 monclient: shutdown 2024-03-25T19:38:37.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:37.437+0000 7f1fcf44a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:37.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:37.437+0000 7f1fcf44a640 1 -- 172.21.15.44:0/3217565867 >> v1:172.21.15.111:6789/0 conn(0x7f1fc81430c0 legacy=0x7f1fc81e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:37.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:37.437+0000 7f1fcf44a640 1 -- 172.21.15.44:0/3217565867 shutdown_connections 2024-03-25T19:38:37.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:37.437+0000 7f1fcf44a640 1 -- 172.21.15.44:0/3217565867 >> 172.21.15.44:0/3217565867 conn(0x7f1fc80b7ea0 msgr2=0x7f1fc8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:37.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:37.438+0000 7f1fcf44a640 1 -- 172.21.15.44:0/3217565867 shutdown_connections 2024-03-25T19:38:37.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:37.438+0000 7f1fcf44a640 1 -- 172.21.15.44:0/3217565867 wait complete. 2024-03-25T19:38:37.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:37.438+0000 7f1fcf44a640 1 librados: shutdown 2024-03-25T19:38:37.450 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:39.207 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:03:30.109 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:39.216 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:03:30.118 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:39.246 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:03:30.147 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:39.261 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:03:30.163 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:39.271 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:03:30.173 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:39.280 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:03:30.160 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:39.290 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:03:30.170 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:39.296 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:03:30.198 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:39.331 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:03:30.210 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:44.256 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:44.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.373+0000 7fb2515c6640 10 monclient: get_monmap_and_config 2024-03-25T19:38:44.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.373+0000 7fb2515c6640 10 monclient: build_initial_monmap 2024-03-25T19:38:44.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.373+0000 7fb2515c6640 10 monclient: monmap: 2024-03-25T19:38:44.372 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:38:44.372 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:44.372 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:38:44.373255+0000 2024-03-25T19:38:44.372 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:38:44.373255+0000 2024-03-25T19:38:44.372 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:44.372 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:44.372 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:38:44.373 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:38:44.373 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:38:44.373 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:44.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.373+0000 7fb2515c6640 1 Processor -- start 2024-03-25T19:38:44.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.373+0000 7fb2515c6640 1 -- start start 2024-03-25T19:38:44.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.373+0000 7fb2515c6640 10 monclient: init 2024-03-25T19:38:44.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.374+0000 7fb2515c6640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:44.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.374+0000 7fb2515c6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:38:44.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.374+0000 7fb2515c6640 10 monclient(hunting): picked mon.noname-b con 0x7fb24c0a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T19:38:44.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.374+0000 7fb2515c6640 10 monclient(hunting): picked mon.noname-c con 0x7fb24c13f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:38:44.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.374+0000 7fb2515c6640 10 monclient(hunting): picked mon.noname-a con 0x7fb24c1430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:38:44.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.374+0000 7fb2515c6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c142520 con 0x7fb24c1430c0 2024-03-25T19:38:44.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.374+0000 7fb2515c6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c0b28a0 con 0x7fb24c13f710 2024-03-25T19:38:44.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.374+0000 7fb2515c6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c0b2a30 con 0x7fb24c0a98f0 2024-03-25T19:38:44.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.374+0000 7fb2515c6640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:44.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.374+0000 7fb2515c6640 10 monclient(hunting): authenticate will time out at 5588.544682s 2024-03-25T19:38:44.381 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.382+0000 7fb24affd640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fb24c0a98f0 0x7fb24c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:40194/0 (socket says 172.21.15.44:40194) 2024-03-25T19:38:44.381 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.382+0000 7fb24affd640 1 -- 172.21.15.44:0/639540507 learned_addr learned my addr 172.21.15.44:0/639540507 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:38:44.392 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.393+0000 7fb249ffb640 1 -- 172.21.15.44:0/639540507 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1668538156 0 0) 0x7fb24c0b28a0 con 0x7fb24c13f710 2024-03-25T19:38:44.392 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.393+0000 7fb249ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:44.392 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.393+0000 7fb249ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:44.392 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.393+0000 7fb249ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:44.393 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.393+0000 7fb249ffb640 10 monclient(hunting): my global_id is 4363 2024-03-25T19:38:44.393 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.393+0000 7fb249ffb640 1 -- 172.21.15.44:0/639540507 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb234003520 con 0x7fb24c13f710 2024-03-25T19:38:44.394 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.395+0000 7fb249ffb640 1 -- 172.21.15.44:0/639540507 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700789579 0 0) 0x7fb24c142520 con 0x7fb24c1430c0 2024-03-25T19:38:44.394 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.395+0000 7fb249ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:44.394 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.395+0000 7fb249ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:44.394 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.395+0000 7fb249ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:44.395 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.395+0000 7fb249ffb640 10 monclient(hunting): my global_id is 4383 2024-03-25T19:38:44.395 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.395+0000 7fb249ffb640 1 -- 172.21.15.44:0/639540507 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb24c0b28a0 con 0x7fb24c1430c0 2024-03-25T19:38:44.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.406+0000 7fb249ffb640 1 -- 172.21.15.44:0/639540507 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1893828479 0 0) 0x7fb234003520 con 0x7fb24c13f710 2024-03-25T19:38:44.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.406+0000 7fb249ffb640 1 -- 172.21.15.44:0/639540507 >> v1:172.21.15.44:6790/0 conn(0x7fb24c0a98f0 legacy=0x7fb24c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:44.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.406+0000 7fb249ffb640 1 -- 172.21.15.44:0/639540507 >> v1:172.21.15.44:6789/0 conn(0x7fb24c1430c0 legacy=0x7fb24c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:44.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.406+0000 7fb249ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:38:44.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.406+0000 7fb249ffb640 1 monclient: found mon.noname-c 2024-03-25T19:38:44.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.406+0000 7fb249ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:44.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.406+0000 7fb249ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:38:44.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.406+0000 7fb249ffb640 1 -- 172.21.15.44:0/639540507 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24c0b44c0 con 0x7fb24c13f710 2024-03-25T19:38:44.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.406+0000 7fb249ffb640 10 monclient: _finish_auth 0 2024-03-25T19:38:44.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.406+0000 7fb249ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:38:44.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.406+0000 7fb249ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:44.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.406+0000 7fb249ffb640 1 -- 172.21.15.44:0/639540507 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fb240001690 con 0x7fb24c13f710 2024-03-25T19:38:44.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.406+0000 7fb249ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:44.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.406+0000 7fb249ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:44.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.406+0000 7fb249ffb640 10 monclient: dump: 2024-03-25T19:38:44.406 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:44.407 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:44.407 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:44.407 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:44.407 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:44.407 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:44.407 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:44.407 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:44.407 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:44.407 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:44.407 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.406+0000 7fb2515c6640 5 monclient: authenticate success, global_id 4363 2024-03-25T19:38:44.407 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.406+0000 7fb2515c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:44.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.415+0000 7fb249ffb640 1 -- 172.21.15.44:0/639540507 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2400017f0 con 0x7fb24c13f710 2024-03-25T19:38:44.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.415+0000 7fb249ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:44.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.415+0000 7fb249ffb640 1 -- 172.21.15.44:0/639540507 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fb240001c00 con 0x7fb24c13f710 2024-03-25T19:38:44.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.415+0000 7fb249ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:44.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.415+0000 7fb249ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:44.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.415+0000 7fb249ffb640 10 monclient: dump: 2024-03-25T19:38:44.414 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:44.414 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:44.414 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:44.414 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:44.415 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:44.415 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:44.415 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:44.415 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:44.415 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:44.415 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:44.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.415+0000 7fb2515c6640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:44.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.415+0000 7fb2515c6640 4 set_mon_vals no callback set 2024-03-25T19:38:44.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.416+0000 7fb2515c6640 10 monclient: shutdown 2024-03-25T19:38:44.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.416+0000 7fb2515c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:44.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.416+0000 7fb2515c6640 1 -- 172.21.15.44:0/639540507 >> v1:172.21.15.111:6789/0 conn(0x7fb24c13f710 legacy=0x7fb24c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:44.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.416+0000 7fb2515c6640 1 -- 172.21.15.44:0/639540507 shutdown_connections 2024-03-25T19:38:44.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.416+0000 7fb2515c6640 1 -- 172.21.15.44:0/639540507 >> 172.21.15.44:0/639540507 conn(0x7fb24c0b7ea0 msgr2=0x7fb24c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:44.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.416+0000 7fb2515c6640 1 -- 172.21.15.44:0/639540507 shutdown_connections 2024-03-25T19:38:44.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.416+0000 7fb2515c6640 1 -- 172.21.15.44:0/639540507 wait complete. 2024-03-25T19:38:44.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.416+0000 7fb2515c6640 10 monclient: build_initial_monmap 2024-03-25T19:38:44.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.417+0000 7fb2515c6640 10 monclient: monmap: 2024-03-25T19:38:44.418 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:38:44.418 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:44.418 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:38:44.418 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:38:44.418 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:44.418 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:38:44.418 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:38:44.418 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:38:44.418 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:38:44.418 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:44.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.417+0000 7fb2515c6640 1 librados: starting msgr at 2024-03-25T19:38:44.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.417+0000 7fb2515c6640 1 librados: starting objecter 2024-03-25T19:38:44.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.417+0000 7fb2515c6640 1 Processor -- start 2024-03-25T19:38:44.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.417+0000 7fb2515c6640 1 -- start start 2024-03-25T19:38:44.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.417+0000 7fb2515c6640 1 librados: setting wanted keys 2024-03-25T19:38:44.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.417+0000 7fb2515c6640 1 librados: calling monclient init 2024-03-25T19:38:44.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.417+0000 7fb2515c6640 10 monclient: init 2024-03-25T19:38:44.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.418+0000 7fb2515c6640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:44.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.418+0000 7fb2515c6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:38:44.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.418+0000 7fb2515c6640 10 monclient(hunting): picked mon.noname-a con 0x7fb24c0a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:38:44.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.418+0000 7fb2515c6640 10 monclient(hunting): picked mon.noname-b con 0x7fb24c13f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:38:44.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.418+0000 7fb2515c6640 10 monclient(hunting): picked mon.noname-c con 0x7fb24c1430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:38:44.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.418+0000 7fb2515c6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c1d7490 con 0x7fb24c0a98f0 2024-03-25T19:38:44.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.418+0000 7fb2515c6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c1d76c0 con 0x7fb24c13f710 2024-03-25T19:38:44.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.418+0000 7fb2515c6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c1d82f0 con 0x7fb24c1430c0 2024-03-25T19:38:44.420 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.418+0000 7fb2515c6640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:44.420 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.418+0000 7fb2515c6640 10 monclient(hunting): authenticate will time out at 5588.588246s 2024-03-25T19:38:44.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.427+0000 7fb24b7fe640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fb24c1430c0 0x7fb24c133fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:40228/0 (socket says 172.21.15.44:40228) 2024-03-25T19:38:44.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.427+0000 7fb24b7fe640 1 -- 172.21.15.44:0/2727380243 learned_addr learned my addr 172.21.15.44:0/2727380243 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:38:44.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.428+0000 7fb2337fe640 10 client.?.objecter ms_handle_connect 0x7fb24c13f710 2024-03-25T19:38:44.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.428+0000 7fb2337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:44.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.429+0000 7fb2337fe640 10 client.?.objecter ms_handle_connect 0x7fb24c1430c0 2024-03-25T19:38:44.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.429+0000 7fb2337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:44.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.439+0000 7fb2337fe640 1 -- 172.21.15.44:0/2727380243 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1844755275 0 0) 0x7fb24c1d76c0 con 0x7fb24c13f710 2024-03-25T19:38:44.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.439+0000 7fb2337fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:44.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.439+0000 7fb2337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:44.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.439+0000 7fb2337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:44.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.439+0000 7fb2337fe640 10 monclient(hunting): my global_id is 4366 2024-03-25T19:38:44.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.439+0000 7fb2337fe640 1 -- 172.21.15.44:0/2727380243 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb238004eb0 con 0x7fb24c13f710 2024-03-25T19:38:44.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.440+0000 7fb2337fe640 1 -- 172.21.15.44:0/2727380243 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2473748750 0 0) 0x7fb24c1d82f0 con 0x7fb24c1430c0 2024-03-25T19:38:44.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.440+0000 7fb2337fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:44.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.440+0000 7fb2337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:44.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.440+0000 7fb2337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:44.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.440+0000 7fb2337fe640 10 monclient(hunting): my global_id is 4355 2024-03-25T19:38:44.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.440+0000 7fb2337fe640 1 -- 172.21.15.44:0/2727380243 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb24c1d76c0 con 0x7fb24c1430c0 2024-03-25T19:38:44.445 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.446+0000 7fb2337fe640 10 client.?.objecter ms_handle_connect 0x7fb24c0a98f0 2024-03-25T19:38:44.445 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.447+0000 7fb2337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:44.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.448+0000 7fb2337fe640 1 -- 172.21.15.44:0/2727380243 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1988356540 0 0) 0x7fb238004eb0 con 0x7fb24c13f710 2024-03-25T19:38:44.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.448+0000 7fb2337fe640 1 -- 172.21.15.44:0/2727380243 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb24c1d82f0 con 0x7fb24c13f710 2024-03-25T19:38:44.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.448+0000 7fb2337fe640 1 -- 172.21.15.44:0/2727380243 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fb240001bc0 con 0x7fb24c13f710 2024-03-25T19:38:44.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.448+0000 7fb2337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:44.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.448+0000 7fb2337fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:44.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.448+0000 7fb2337fe640 10 monclient(hunting): dump: 2024-03-25T19:38:44.448 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:44.448 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:44.448 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:44.448 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:44.448 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:44.448 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:44.448 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:44.448 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:44.448 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:44.448 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:44.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.448+0000 7fb2337fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:44.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.459+0000 7fb2337fe640 1 -- 172.21.15.44:0/2727380243 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3449820138 0 0) 0x7fb24c1d76c0 con 0x7fb24c1430c0 2024-03-25T19:38:44.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.459+0000 7fb2337fe640 1 -- 172.21.15.44:0/2727380243 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb238004eb0 con 0x7fb24c1430c0 2024-03-25T19:38:44.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.459+0000 7fb2337fe640 1 -- 172.21.15.44:0/2727380243 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fb244001c50 con 0x7fb24c1430c0 2024-03-25T19:38:44.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.459+0000 7fb2337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:44.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.459+0000 7fb2337fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:44.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.459+0000 7fb2337fe640 10 monclient(hunting): dump: 2024-03-25T19:38:44.459 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:44.459 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:44.459 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:44.459 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:44.459 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:44.459 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:44.459 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:44.459 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:44.459 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:44.459 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:44.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.464+0000 7fb2337fe640 1 -- 172.21.15.44:0/2727380243 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 204290110 0 0) 0x7fb24c1d82f0 con 0x7fb24c13f710 2024-03-25T19:38:44.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.464+0000 7fb2337fe640 1 -- 172.21.15.44:0/2727380243 >> v1:172.21.15.44:6790/0 conn(0x7fb24c1430c0 legacy=0x7fb24c133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:44.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.464+0000 7fb2337fe640 1 -- 172.21.15.44:0/2727380243 >> v1:172.21.15.44:6789/0 conn(0x7fb24c0a98f0 legacy=0x7fb24c1d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:44.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.464+0000 7fb2337fe640 10 monclient: _finish_hunting 0 2024-03-25T19:38:44.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.464+0000 7fb2337fe640 1 monclient: found mon.b 2024-03-25T19:38:44.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.464+0000 7fb2337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:44.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.464+0000 7fb2337fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:44.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.464+0000 7fb2337fe640 1 -- 172.21.15.44:0/2727380243 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24c138af0 con 0x7fb24c13f710 2024-03-25T19:38:44.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.464+0000 7fb2337fe640 10 monclient: _finish_auth 0 2024-03-25T19:38:44.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.464+0000 7fb2337fe640 10 monclient: _check_auth_tickets 2024-03-25T19:38:44.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.464+0000 7fb2337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:44.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.464+0000 7fb2515c6640 5 monclient: authenticate success, global_id 4366 2024-03-25T19:38:44.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.465+0000 7fb2515c6640 10 monclient: _renew_subs 2024-03-25T19:38:44.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.465+0000 7fb2515c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:44.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.465+0000 7fb2515c6640 1 -- 172.21.15.44:0/2727380243 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb24c1376d0 con 0x7fb24c13f710 2024-03-25T19:38:44.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.465+0000 7fb2515c6640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:44.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.465+0000 7fb2515c6640 10 monclient: _renew_subs 2024-03-25T19:38:44.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.465+0000 7fb2515c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:44.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.465+0000 7fb2515c6640 1 -- 172.21.15.44:0/2727380243 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb24c134fc0 con 0x7fb24c13f710 2024-03-25T19:38:44.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.465+0000 7fb2515c6640 1 librados: init done 2024-03-25T19:38:44.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.466+0000 7fb2515c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:44.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.466+0000 7fb2515c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:44.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.466+0000 7fb2515c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:44.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.466+0000 7fb2515c6640 1 -- 172.21.15.44:0/2727380243 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb24c1d7850 con 0x7fb24c13f710 2024-03-25T19:38:44.483 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.485+0000 7fb2337fe640 1 -- 172.21.15.44:0/2727380243 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2400038a0 con 0x7fb24c13f710 2024-03-25T19:38:44.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.485+0000 7fb2337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:44.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.485+0000 7fb2337fe640 1 -- 172.21.15.44:0/2727380243 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fb240001250 con 0x7fb24c13f710 2024-03-25T19:38:44.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.485+0000 7fb2337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:44.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.485+0000 7fb2337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:44.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.485+0000 7fb2337fe640 10 monclient: dump: 2024-03-25T19:38:44.484 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:44.484 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:44.484 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:44.484 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:44.484 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:44.484 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:44.484 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:44.484 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:44.484 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:44.484 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:44.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.485+0000 7fb2497fa640 4 set_mon_vals no callback set 2024-03-25T19:38:44.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.486+0000 7fb2337fe640 1 -- 172.21.15.44:0/2727380243 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7fb240019ae0 con 0x7fb24c13f710 2024-03-25T19:38:44.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.486+0000 7fb24affd640 10 monclient: get_auth_request con 0x7fb23806de90 auth_method 0 2024-03-25T19:38:44.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.486+0000 7fb2337fe640 10 client.4366.objecter ms_handle_connect 0x7fb23806de90 2024-03-25T19:38:44.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.488+0000 7fb2337fe640 1 -- 172.21.15.44:0/2727380243 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7fb240053740 con 0x7fb24c13f710 2024-03-25T19:38:44.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.488+0000 7fb2337fe640 10 client.4366.objecter ms_dispatch 0x7fb24c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:44.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.488+0000 7fb2337fe640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:44.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.488+0000 7fb2337fe640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:44.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.488+0000 7fb2337fe640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-25T19:38:44.737 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.738+0000 7fb2337fe640 1 -- 172.21.15.44:0/2727380243 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb24004f080 con 0x7fb24c13f710 2024-03-25T19:38:44.737 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.738+0000 7fb2337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:44.737 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.738+0000 7fb2337fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:44.990 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.991+0000 7fb2515c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:44.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.991+0000 7fb2515c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:44.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.991+0000 7fb2515c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:44.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:44.991+0000 7fb2515c6640 1 -- 172.21.15.44:0/2727380243 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb24c14f300 con 0x7fb24c13f710 2024-03-25T19:38:45.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:45.127+0000 7fb2337fe640 1 -- 172.21.15.44:0/2727380243 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7fb240051180 con 0x7fb24c13f710 2024-03-25T19:38:45.127 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:45.127+0000 7fb2337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:45.127 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:45.127+0000 7fb2337fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:45.127 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:38:45.127 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:45.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:45.129+0000 7fb2515c6640 10 librados: watch_flush enter 2024-03-25T19:38:45.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:45.129+0000 7fb2515c6640 10 librados: watch_flush exit 2024-03-25T19:38:45.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:45.129+0000 7fb2515c6640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-25T19:38:45.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:45.129+0000 7fb2515c6640 10 client.4366.objecter successfully canceled tick 2024-03-25T19:38:45.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:45.130+0000 7fb2515c6640 1 -- 172.21.15.44:0/2727380243 >> v1:172.21.15.111:6800/3883215793 conn(0x7fb23806de90 legacy=0x7fb238070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:45.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:45.130+0000 7fb2515c6640 10 monclient: shutdown 2024-03-25T19:38:45.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:45.130+0000 7fb2515c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:45.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:45.130+0000 7fb2515c6640 1 -- 172.21.15.44:0/2727380243 >> v1:172.21.15.111:6789/0 conn(0x7fb24c13f710 legacy=0x7fb24c1383e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:45.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:45.130+0000 7fb2515c6640 1 -- 172.21.15.44:0/2727380243 shutdown_connections 2024-03-25T19:38:45.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:45.130+0000 7fb2515c6640 1 -- 172.21.15.44:0/2727380243 >> 172.21.15.44:0/2727380243 conn(0x7fb24c0b7ea0 msgr2=0x7fb24c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:45.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:45.130+0000 7fb2515c6640 1 -- 172.21.15.44:0/2727380243 shutdown_connections 2024-03-25T19:38:45.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:45.130+0000 7fb2515c6640 1 -- 172.21.15.44:0/2727380243 wait complete. 2024-03-25T19:38:45.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:45.130+0000 7fb2515c6640 1 librados: shutdown 2024-03-25T19:38:45.142 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:51.948 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:52.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.063+0000 7f852068d640 10 monclient: get_monmap_and_config 2024-03-25T19:38:52.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.063+0000 7f852068d640 10 monclient: build_initial_monmap 2024-03-25T19:38:52.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.063+0000 7f852068d640 10 monclient: monmap: 2024-03-25T19:38:52.064 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:38:52.064 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:52.064 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:38:52.064928+0000 2024-03-25T19:38:52.064 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:38:52.064928+0000 2024-03-25T19:38:52.064 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:52.064 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:52.064 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:38:52.064 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:38:52.064 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:38:52.064 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:52.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.064+0000 7f852068d640 1 Processor -- start 2024-03-25T19:38:52.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.064+0000 7f852068d640 1 -- start start 2024-03-25T19:38:52.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.064+0000 7f852068d640 10 monclient: init 2024-03-25T19:38:52.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.065+0000 7f852068d640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:52.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.065+0000 7f852068d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:38:52.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.065+0000 7f852068d640 10 monclient(hunting): picked mon.noname-a con 0x7f85180a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:38:52.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.065+0000 7f852068d640 10 monclient(hunting): picked mon.noname-b con 0x7f851813f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:38:52.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.065+0000 7f852068d640 10 monclient(hunting): picked mon.noname-c con 0x7f85181430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:38:52.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.065+0000 7f852068d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8518142520 con 0x7f85180a98f0 2024-03-25T19:38:52.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.066+0000 7f852068d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85180b28a0 con 0x7f85181430c0 2024-03-25T19:38:52.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.066+0000 7f852068d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85180b2a30 con 0x7f851813f710 2024-03-25T19:38:52.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.066+0000 7f852068d640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:52.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.066+0000 7f852068d640 10 monclient(hunting): authenticate will time out at 5596.236891s 2024-03-25T19:38:52.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.073+0000 7f851ec2a640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f85181430c0 0x7f851814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:50108/0 (socket says 172.21.15.44:50108) 2024-03-25T19:38:52.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.073+0000 7f851ec2a640 1 -- 172.21.15.44:0/1853319982 learned_addr learned my addr 172.21.15.44:0/1853319982 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:38:52.093 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.093+0000 7f851d427640 1 -- 172.21.15.44:0/1853319982 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928362486 0 0) 0x7f85180b28a0 con 0x7f85181430c0 2024-03-25T19:38:52.093 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.093+0000 7f851d427640 10 monclient(hunting): _negotiate 2024-03-25T19:38:52.093 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.093+0000 7f851d427640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:52.093 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.093+0000 7f851d427640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:52.093 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.093+0000 7f851d427640 10 monclient(hunting): my global_id is 4369 2024-03-25T19:38:52.093 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.093+0000 7f851d427640 1 -- 172.21.15.44:0/1853319982 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8504003520 con 0x7f85181430c0 2024-03-25T19:38:52.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.106+0000 7f851d427640 1 -- 172.21.15.44:0/1853319982 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1170656400 0 0) 0x7f8504003520 con 0x7f85181430c0 2024-03-25T19:38:52.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.106+0000 7f851d427640 1 -- 172.21.15.44:0/1853319982 >> v1:172.21.15.44:6790/0 conn(0x7f851813f710 legacy=0x7f85181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:52.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.106+0000 7f851d427640 1 -- 172.21.15.44:0/1853319982 >> v1:172.21.15.44:6789/0 conn(0x7f85180a98f0 legacy=0x7f85180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:52.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.106+0000 7f851d427640 10 monclient: _finish_hunting 0 2024-03-25T19:38:52.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.106+0000 7f851d427640 1 monclient: found mon.noname-c 2024-03-25T19:38:52.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.106+0000 7f851d427640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:52.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.106+0000 7f851d427640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:38:52.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.106+0000 7f851d427640 1 -- 172.21.15.44:0/1853319982 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85180b44c0 con 0x7f85181430c0 2024-03-25T19:38:52.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.106+0000 7f851d427640 10 monclient: _finish_auth 0 2024-03-25T19:38:52.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.106+0000 7f851d427640 10 monclient: _check_auth_tickets 2024-03-25T19:38:52.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.106+0000 7f851d427640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:52.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.106+0000 7f851d427640 1 -- 172.21.15.44:0/1853319982 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f8510001a10 con 0x7f85181430c0 2024-03-25T19:38:52.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.106+0000 7f851d427640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:52.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.106+0000 7f851d427640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:52.107 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.106+0000 7f851d427640 10 monclient: dump: 2024-03-25T19:38:52.107 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:52.108 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:52.108 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:52.108 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:52.108 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:52.108 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:52.108 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:52.108 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:52.108 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:52.108 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:52.108 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.106+0000 7f852068d640 5 monclient: authenticate success, global_id 4369 2024-03-25T19:38:52.108 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.106+0000 7f852068d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:52.125 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.125+0000 7f851d427640 1 -- 172.21.15.44:0/1853319982 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8510001b70 con 0x7f85181430c0 2024-03-25T19:38:52.125 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.125+0000 7f851d427640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:52.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.125+0000 7f851d427640 1 -- 172.21.15.44:0/1853319982 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f8510001f80 con 0x7f85181430c0 2024-03-25T19:38:52.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.125+0000 7f852068d640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:52.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.125+0000 7f852068d640 4 set_mon_vals no callback set 2024-03-25T19:38:52.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.125+0000 7f851d427640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:52.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.125+0000 7f851d427640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:52.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.125+0000 7f851d427640 10 monclient: dump: 2024-03-25T19:38:52.126 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:52.126 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:52.126 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:52.126 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:52.126 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:52.126 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:52.126 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:52.126 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:52.127 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:52.127 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:52.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.126+0000 7f852068d640 10 monclient: shutdown 2024-03-25T19:38:52.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.126+0000 7f852068d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:52.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.126+0000 7f852068d640 1 -- 172.21.15.44:0/1853319982 >> v1:172.21.15.111:6789/0 conn(0x7f85181430c0 legacy=0x7f851814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:52.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.126+0000 7f852068d640 1 -- 172.21.15.44:0/1853319982 shutdown_connections 2024-03-25T19:38:52.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.126+0000 7f852068d640 1 -- 172.21.15.44:0/1853319982 >> 172.21.15.44:0/1853319982 conn(0x7f85180b7ea0 msgr2=0x7f85180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:52.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.126+0000 7f852068d640 1 -- 172.21.15.44:0/1853319982 shutdown_connections 2024-03-25T19:38:52.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.126+0000 7f852068d640 1 -- 172.21.15.44:0/1853319982 wait complete. 2024-03-25T19:38:52.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.127+0000 7f852068d640 10 monclient: build_initial_monmap 2024-03-25T19:38:52.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.127+0000 7f852068d640 10 monclient: monmap: 2024-03-25T19:38:52.129 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:38:52.129 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:52.129 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:38:52.129 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:38:52.129 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:52.130 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:38:52.130 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:38:52.130 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:38:52.130 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:38:52.130 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:52.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.127+0000 7f852068d640 1 librados: starting msgr at 2024-03-25T19:38:52.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.127+0000 7f852068d640 1 librados: starting objecter 2024-03-25T19:38:52.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.127+0000 7f852068d640 1 Processor -- start 2024-03-25T19:38:52.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.128+0000 7f852068d640 1 -- start start 2024-03-25T19:38:52.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.128+0000 7f852068d640 1 librados: setting wanted keys 2024-03-25T19:38:52.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.128+0000 7f852068d640 1 librados: calling monclient init 2024-03-25T19:38:52.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.128+0000 7f852068d640 10 monclient: init 2024-03-25T19:38:52.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.128+0000 7f852068d640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:52.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.128+0000 7f852068d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:38:52.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.128+0000 7f852068d640 10 monclient(hunting): picked mon.noname-b con 0x7f85180a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:38:52.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.129+0000 7f852068d640 10 monclient(hunting): picked mon.noname-c con 0x7f851813f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:38:52.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.129+0000 7f852068d640 10 monclient(hunting): picked mon.noname-a con 0x7f85181430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:38:52.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.129+0000 7f852068d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85181396e0 con 0x7f85181430c0 2024-03-25T19:38:52.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.129+0000 7f852068d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8518138530 con 0x7f85180a98f0 2024-03-25T19:38:52.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.129+0000 7f852068d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85181399c0 con 0x7f851813f710 2024-03-25T19:38:52.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.129+0000 7f852068d640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:52.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.129+0000 7f852068d640 10 monclient(hunting): authenticate will time out at 5596.299917s 2024-03-25T19:38:52.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.145+0000 7f851e429640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f85180a98f0 0x7f8518137820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:50170/0 (socket says 172.21.15.44:50170) 2024-03-25T19:38:52.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.145+0000 7f851e429640 1 -- 172.21.15.44:0/3855857039 learned_addr learned my addr 172.21.15.44:0/3855857039 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:38:52.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.147+0000 7f850effd640 10 client.?.objecter ms_handle_connect 0x7f85180a98f0 2024-03-25T19:38:52.147 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.147+0000 7f850effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:52.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.151+0000 7f850effd640 10 client.?.objecter ms_handle_connect 0x7f851813f710 2024-03-25T19:38:52.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.151+0000 7f850effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:52.154 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.155+0000 7f850effd640 10 client.?.objecter ms_handle_connect 0x7f85181430c0 2024-03-25T19:38:52.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.155+0000 7f850effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:52.160 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.160+0000 7f850effd640 1 -- 172.21.15.44:0/3855857039 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894616579 0 0) 0x7f8518138530 con 0x7f85180a98f0 2024-03-25T19:38:52.160 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.160+0000 7f850effd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:52.160 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.160+0000 7f850effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:52.160 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.160+0000 7f850effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:52.160 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.160+0000 7f850effd640 10 monclient(hunting): my global_id is 4372 2024-03-25T19:38:52.160 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.160+0000 7f850effd640 1 -- 172.21.15.44:0/3855857039 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84fc004eb0 con 0x7f85180a98f0 2024-03-25T19:38:52.162 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.162+0000 7f850effd640 1 -- 172.21.15.44:0/3855857039 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 925781872 0 0) 0x7f85181396e0 con 0x7f85181430c0 2024-03-25T19:38:52.162 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.162+0000 7f850effd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:52.162 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.162+0000 7f850effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:52.162 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.162+0000 7f850effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:52.162 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.162+0000 7f850effd640 10 monclient(hunting): my global_id is 4392 2024-03-25T19:38:52.162 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.162+0000 7f850effd640 1 -- 172.21.15.44:0/3855857039 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8518138530 con 0x7f85181430c0 2024-03-25T19:38:52.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.165+0000 7f850effd640 1 -- 172.21.15.44:0/3855857039 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1455201698 0 0) 0x7f85181399c0 con 0x7f851813f710 2024-03-25T19:38:52.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.165+0000 7f850effd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:52.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.165+0000 7f850effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:52.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.165+0000 7f850effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:52.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.165+0000 7f850effd640 10 monclient(hunting): my global_id is 4361 2024-03-25T19:38:52.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.165+0000 7f850effd640 1 -- 172.21.15.44:0/3855857039 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85181396e0 con 0x7f851813f710 2024-03-25T19:38:52.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.177+0000 7f850effd640 1 -- 172.21.15.44:0/3855857039 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2438668990 0 0) 0x7f84fc004eb0 con 0x7f85180a98f0 2024-03-25T19:38:52.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.177+0000 7f850effd640 1 -- 172.21.15.44:0/3855857039 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85181399c0 con 0x7f85180a98f0 2024-03-25T19:38:52.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.177+0000 7f850effd640 1 -- 172.21.15.44:0/3855857039 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f8514001a30 con 0x7f85180a98f0 2024-03-25T19:38:52.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.177+0000 7f850effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:52.178 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.177+0000 7f850effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:52.178 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.177+0000 7f850effd640 10 monclient(hunting): dump: 2024-03-25T19:38:52.178 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:52.178 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:52.178 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:52.178 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:52.178 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:52.178 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:52.178 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:52.178 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:52.179 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:52.179 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:52.179 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.177+0000 7f850effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:52.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.182+0000 7f850effd640 1 -- 172.21.15.44:0/3855857039 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3915625391 0 0) 0x7f85181396e0 con 0x7f851813f710 2024-03-25T19:38:52.184 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.182+0000 7f850effd640 1 -- 172.21.15.44:0/3855857039 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84fc004eb0 con 0x7f851813f710 2024-03-25T19:38:52.184 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.182+0000 7f850effd640 1 -- 172.21.15.44:0/3855857039 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f8508001860 con 0x7f851813f710 2024-03-25T19:38:52.184 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.182+0000 7f850effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:52.184 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.182+0000 7f850effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:52.184 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.182+0000 7f850effd640 10 monclient(hunting): dump: 2024-03-25T19:38:52.184 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:52.184 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:52.184 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:52.184 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:52.184 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:52.184 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:52.184 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:52.184 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:52.185 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:52.185 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:52.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.182+0000 7f850effd640 1 -- 172.21.15.44:0/3855857039 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1444008792 0 0) 0x7f85181399c0 con 0x7f85180a98f0 2024-03-25T19:38:52.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.182+0000 7f850effd640 1 -- 172.21.15.44:0/3855857039 >> v1:172.21.15.44:6790/0 conn(0x7f851813f710 legacy=0x7f85181da850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:52.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.182+0000 7f850effd640 1 -- 172.21.15.44:0/3855857039 >> v1:172.21.15.44:6789/0 conn(0x7f85181430c0 legacy=0x7f85181e1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:52.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.182+0000 7f850effd640 10 monclient: _finish_hunting 0 2024-03-25T19:38:52.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.182+0000 7f850effd640 1 monclient: found mon.b 2024-03-25T19:38:52.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.182+0000 7f850effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:52.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.182+0000 7f850effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:52.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.182+0000 7f850effd640 1 -- 172.21.15.44:0/3855857039 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8518138a50 con 0x7f85180a98f0 2024-03-25T19:38:52.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.182+0000 7f850effd640 10 monclient: _finish_auth 0 2024-03-25T19:38:52.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.182+0000 7f850effd640 10 monclient: _check_auth_tickets 2024-03-25T19:38:52.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.182+0000 7f850effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:52.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.182+0000 7f852068d640 5 monclient: authenticate success, global_id 4372 2024-03-25T19:38:52.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.182+0000 7f852068d640 10 monclient: _renew_subs 2024-03-25T19:38:52.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.182+0000 7f852068d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:52.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.182+0000 7f852068d640 1 -- 172.21.15.44:0/3855857039 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85181380a0 con 0x7f85180a98f0 2024-03-25T19:38:52.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.182+0000 7f852068d640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:52.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.183+0000 7f852068d640 10 monclient: _renew_subs 2024-03-25T19:38:52.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.183+0000 7f852068d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:52.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.183+0000 7f852068d640 1 -- 172.21.15.44:0/3855857039 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85181db140 con 0x7f85180a98f0 2024-03-25T19:38:52.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.183+0000 7f852068d640 1 librados: init done 2024-03-25T19:38:52.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.183+0000 7f852068d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:52.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.183+0000 7f852068d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:52.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.183+0000 7f852068d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:52.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.183+0000 7f852068d640 1 -- 172.21.15.44:0/3855857039 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8518138ca0 con 0x7f85180a98f0 2024-03-25T19:38:52.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.455+0000 7f850effd640 1 -- 172.21.15.44:0/3855857039 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8514001c70 con 0x7f85180a98f0 2024-03-25T19:38:52.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.455+0000 7f850effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:52.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.455+0000 7f850effd640 1 -- 172.21.15.44:0/3855857039 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f85140038c0 con 0x7f85180a98f0 2024-03-25T19:38:52.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.455+0000 7f850effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:52.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.455+0000 7f850effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:52.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.455+0000 7f850effd640 10 monclient: dump: 2024-03-25T19:38:52.456 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:52.456 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:52.456 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:52.456 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:52.456 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:52.456 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:52.456 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:52.456 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:52.456 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:52.456 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:52.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.455+0000 7f851cc26640 4 set_mon_vals no callback set 2024-03-25T19:38:52.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.456+0000 7f850effd640 1 -- 172.21.15.44:0/3855857039 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f8514003aa0 con 0x7f85180a98f0 2024-03-25T19:38:52.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.456+0000 7f851dc28640 10 monclient: get_auth_request con 0x7f84fc06df20 auth_method 0 2024-03-25T19:38:52.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.457+0000 7f850effd640 10 client.4372.objecter ms_handle_connect 0x7f84fc06df20 2024-03-25T19:38:52.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.459+0000 7f850effd640 1 -- 172.21.15.44:0/3855857039 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f8514054780 con 0x7f85180a98f0 2024-03-25T19:38:52.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.459+0000 7f850effd640 10 client.4372.objecter ms_dispatch 0x7f8518000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:52.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.459+0000 7f850effd640 3 client.4372.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:52.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.459+0000 7f850effd640 3 client.4372.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:52.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.459+0000 7f850effd640 20 client.4372.objecter dump_active .. 0 homeless 2024-03-25T19:38:52.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.461+0000 7f850effd640 1 -- 172.21.15.44:0/3855857039 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8514050290 con 0x7f85180a98f0 2024-03-25T19:38:52.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.462+0000 7f850effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:52.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.462+0000 7f850effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:52.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.690+0000 7f852068d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:52.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.690+0000 7f852068d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:52.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.690+0000 7f852068d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:38:52.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.690+0000 7f852068d640 1 -- 172.21.15.44:0/3855857039 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85181386c0 con 0x7f85180a98f0 2024-03-25T19:38:52.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.816+0000 7f850effd640 1 -- 172.21.15.44:0/3855857039 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f85140521c0 con 0x7f85180a98f0 2024-03-25T19:38:52.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.816+0000 7f850effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:52.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.816+0000 7f850effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:52.817 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:38:52.817 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:52.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.819+0000 7f852068d640 10 librados: watch_flush enter 2024-03-25T19:38:52.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.819+0000 7f852068d640 10 librados: watch_flush exit 2024-03-25T19:38:52.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.819+0000 7f852068d640 20 client.4372.objecter shutdown clearing up homeless session... 2024-03-25T19:38:52.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.819+0000 7f852068d640 10 client.4372.objecter successfully canceled tick 2024-03-25T19:38:52.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.819+0000 7f852068d640 1 -- 172.21.15.44:0/3855857039 >> v1:172.21.15.111:6800/3883215793 conn(0x7f84fc06df20 legacy=0x7f84fc0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:52.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.819+0000 7f852068d640 10 monclient: shutdown 2024-03-25T19:38:52.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.819+0000 7f852068d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:52.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.819+0000 7f852068d640 1 -- 172.21.15.44:0/3855857039 >> v1:172.21.15.111:6789/0 conn(0x7f85180a98f0 legacy=0x7f8518137820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:52.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.819+0000 7f852068d640 1 -- 172.21.15.44:0/3855857039 shutdown_connections 2024-03-25T19:38:52.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.819+0000 7f852068d640 1 -- 172.21.15.44:0/3855857039 >> 172.21.15.44:0/3855857039 conn(0x7f85180b7ea0 msgr2=0x7f8518074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:52.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.819+0000 7f852068d640 1 -- 172.21.15.44:0/3855857039 shutdown_connections 2024-03-25T19:38:52.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.819+0000 7f852068d640 1 -- 172.21.15.44:0/3855857039 wait complete. 2024-03-25T19:38:52.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:52.819+0000 7f852068d640 1 librados: shutdown 2024-03-25T19:38:52.831 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:59.217 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:03:50.118 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:59.220 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:03:50.122 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:59.256 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:03:50.158 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:59.287 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:03:50.167 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:59.294 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:03:50.174 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:59.299 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:03:50.201 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:59.302 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:03:50.204 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:59.321 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:03:50.201 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:59.329 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:03:50.231 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:59.639 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:59.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.755+0000 7fa1b7388640 10 monclient: get_monmap_and_config 2024-03-25T19:38:59.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.755+0000 7fa1b7388640 10 monclient: build_initial_monmap 2024-03-25T19:38:59.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.755+0000 7fa1b7388640 10 monclient: monmap: 2024-03-25T19:38:59.755 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:38:59.755 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:59.755 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:38:59.756413+0000 2024-03-25T19:38:59.755 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:38:59.756413+0000 2024-03-25T19:38:59.755 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:59.755 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:59.755 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:38:59.755 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:38:59.755 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:38:59.755 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:59.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.756+0000 7fa1b7388640 1 Processor -- start 2024-03-25T19:38:59.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.756+0000 7fa1b7388640 1 -- start start 2024-03-25T19:38:59.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.756+0000 7fa1b7388640 10 monclient: init 2024-03-25T19:38:59.757 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.757+0000 7fa1b7388640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:59.757 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.757+0000 7fa1b7388640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:38:59.757 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.757+0000 7fa1b7388640 10 monclient(hunting): picked mon.noname-b con 0x7fa1b00a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T19:38:59.757 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.757+0000 7fa1b7388640 10 monclient(hunting): picked mon.noname-c con 0x7fa1b013f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:38:59.757 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.757+0000 7fa1b7388640 10 monclient(hunting): picked mon.noname-a con 0x7fa1b01430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:38:59.757 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.757+0000 7fa1b7388640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b0142520 con 0x7fa1b01430c0 2024-03-25T19:38:59.757 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.757+0000 7fa1b7388640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b00b28a0 con 0x7fa1b013f710 2024-03-25T19:38:59.757 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.757+0000 7fa1b7388640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b00b2a30 con 0x7fa1b00a98f0 2024-03-25T19:38:59.757 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.757+0000 7fa1b7388640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:59.757 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.757+0000 7fa1b7388640 10 monclient(hunting): authenticate will time out at 5603.928174s 2024-03-25T19:38:59.760 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.760+0000 7fa1b5925640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fa1b01430c0 0x7fa1b014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:52250/0 (socket says 172.21.15.44:52250) 2024-03-25T19:38:59.760 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.760+0000 7fa1b5925640 1 -- 172.21.15.44:0/1849171540 learned_addr learned my addr 172.21.15.44:0/1849171540 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:38:59.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.768+0000 7fa1a7fff640 1 -- 172.21.15.44:0/1849171540 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2083664185 0 0) 0x7fa1b0142520 con 0x7fa1b01430c0 2024-03-25T19:38:59.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.768+0000 7fa1a7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:59.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.768+0000 7fa1a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:59.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.768+0000 7fa1a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:59.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.768+0000 7fa1a7fff640 10 monclient(hunting): my global_id is 4395 2024-03-25T19:38:59.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.768+0000 7fa1a7fff640 1 -- 172.21.15.44:0/1849171540 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa19c003520 con 0x7fa1b01430c0 2024-03-25T19:38:59.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.779+0000 7fa1a7fff640 1 -- 172.21.15.44:0/1849171540 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2253323510 0 0) 0x7fa1b00b28a0 con 0x7fa1b013f710 2024-03-25T19:38:59.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.779+0000 7fa1a7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:59.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.779+0000 7fa1a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:59.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.779+0000 7fa1a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:59.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.779+0000 7fa1a7fff640 10 monclient(hunting): my global_id is 4375 2024-03-25T19:38:59.779 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.779+0000 7fa1a7fff640 1 -- 172.21.15.44:0/1849171540 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1b0142520 con 0x7fa1b013f710 2024-03-25T19:38:59.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.791+0000 7fa1a7fff640 1 -- 172.21.15.44:0/1849171540 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212888686 0 0) 0x7fa1b00b2a30 con 0x7fa1b00a98f0 2024-03-25T19:38:59.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.791+0000 7fa1a7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:59.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.791+0000 7fa1a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:59.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.791+0000 7fa1a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:59.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.791+0000 7fa1a7fff640 10 monclient(hunting): my global_id is 4364 2024-03-25T19:38:59.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.791+0000 7fa1a7fff640 1 -- 172.21.15.44:0/1849171540 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1b00b28a0 con 0x7fa1b00a98f0 2024-03-25T19:38:59.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.795+0000 7fa1a7fff640 1 -- 172.21.15.44:0/1849171540 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1444816936 0 0) 0x7fa19c003520 con 0x7fa1b01430c0 2024-03-25T19:38:59.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.795+0000 7fa1a7fff640 1 -- 172.21.15.44:0/1849171540 >> v1:172.21.15.44:6790/0 conn(0x7fa1b00a98f0 legacy=0x7fa1b00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:59.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.795+0000 7fa1a7fff640 1 -- 172.21.15.44:0/1849171540 >> v1:172.21.15.111:6789/0 conn(0x7fa1b013f710 legacy=0x7fa1b01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:59.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.795+0000 7fa1a7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:38:59.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.795+0000 7fa1a7fff640 1 monclient: found mon.noname-a 2024-03-25T19:38:59.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.795+0000 7fa1a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:59.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.795+0000 7fa1a7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:38:59.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.795+0000 7fa1a7fff640 1 -- 172.21.15.44:0/1849171540 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b00b44c0 con 0x7fa1b01430c0 2024-03-25T19:38:59.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.795+0000 7fa1a7fff640 10 monclient: _finish_auth 0 2024-03-25T19:38:59.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.795+0000 7fa1a7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:38:59.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.795+0000 7fa1a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:59.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.795+0000 7fa1a7fff640 1 -- 172.21.15.44:0/1849171540 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fa1a8001a10 con 0x7fa1b01430c0 2024-03-25T19:38:59.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.795+0000 7fa1a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:59.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.795+0000 7fa1a7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:38:59.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.795+0000 7fa1a7fff640 10 monclient: dump: 2024-03-25T19:38:59.797 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:59.797 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:59.797 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:59.797 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:59.797 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:59.797 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:59.797 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:59.797 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:59.797 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:59.797 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:59.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.795+0000 7fa1b7388640 5 monclient: authenticate success, global_id 4395 2024-03-25T19:38:59.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.795+0000 7fa1b7388640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:59.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.823+0000 7fa1a7fff640 1 -- 172.21.15.44:0/1849171540 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1a8001b70 con 0x7fa1b01430c0 2024-03-25T19:38:59.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.823+0000 7fa1a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:59.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.823+0000 7fa1a7fff640 1 -- 172.21.15.44:0/1849171540 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fa1a8001f80 con 0x7fa1b01430c0 2024-03-25T19:38:59.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.823+0000 7fa1b7388640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:59.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.823+0000 7fa1b7388640 4 set_mon_vals no callback set 2024-03-25T19:38:59.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.823+0000 7fa1a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:59.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.823+0000 7fa1a7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:59.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.823+0000 7fa1a7fff640 10 monclient: dump: 2024-03-25T19:38:59.824 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:59.824 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:59.824 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:59.824 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:59.824 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:59.824 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:59.824 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:59.824 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:59.824 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:59.824 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:59.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.824+0000 7fa1b7388640 10 monclient: shutdown 2024-03-25T19:38:59.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.824+0000 7fa1b7388640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:59.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.824+0000 7fa1b7388640 1 -- 172.21.15.44:0/1849171540 >> v1:172.21.15.44:6789/0 conn(0x7fa1b01430c0 legacy=0x7fa1b014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:59.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.824+0000 7fa1b7388640 1 -- 172.21.15.44:0/1849171540 shutdown_connections 2024-03-25T19:38:59.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.824+0000 7fa1b7388640 1 -- 172.21.15.44:0/1849171540 >> 172.21.15.44:0/1849171540 conn(0x7fa1b00b7ea0 msgr2=0x7fa1b00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:59.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.824+0000 7fa1b7388640 1 -- 172.21.15.44:0/1849171540 shutdown_connections 2024-03-25T19:38:59.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.825+0000 7fa1b7388640 1 -- 172.21.15.44:0/1849171540 wait complete. 2024-03-25T19:38:59.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.825+0000 7fa1b7388640 10 monclient: build_initial_monmap 2024-03-25T19:38:59.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.826+0000 7fa1b7388640 10 monclient: monmap: 2024-03-25T19:38:59.826 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:38:59.826 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:59.826 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:38:59.826 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:38:59.826 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:59.826 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:38:59.826 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:38:59.826 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:38:59.826 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:38:59.827 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:59.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.826+0000 7fa1b7388640 1 librados: starting msgr at 2024-03-25T19:38:59.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.826+0000 7fa1b7388640 1 librados: starting objecter 2024-03-25T19:38:59.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.826+0000 7fa1b7388640 1 Processor -- start 2024-03-25T19:38:59.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.826+0000 7fa1b7388640 1 -- start start 2024-03-25T19:38:59.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.826+0000 7fa1b7388640 1 librados: setting wanted keys 2024-03-25T19:38:59.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.826+0000 7fa1b7388640 1 librados: calling monclient init 2024-03-25T19:38:59.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.826+0000 7fa1b7388640 10 monclient: init 2024-03-25T19:38:59.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.827+0000 7fa1b7388640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:59.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.827+0000 7fa1b7388640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:38:59.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.827+0000 7fa1b7388640 10 monclient(hunting): picked mon.noname-b con 0x7fa1b00a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:38:59.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.827+0000 7fa1b7388640 10 monclient(hunting): picked mon.noname-a con 0x7fa1b013f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:38:59.829 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.827+0000 7fa1b7388640 10 monclient(hunting): picked mon.noname-c con 0x7fa1b01430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:38:59.829 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.827+0000 7fa1b7388640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b0136e20 con 0x7fa1b013f710 2024-03-25T19:38:59.829 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.827+0000 7fa1b7388640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b00b7940 con 0x7fa1b00a98f0 2024-03-25T19:38:59.829 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.827+0000 7fa1b7388640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b01351a0 con 0x7fa1b01430c0 2024-03-25T19:38:59.829 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.827+0000 7fa1b7388640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:59.829 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.827+0000 7fa1b7388640 10 monclient(hunting): authenticate will time out at 5603.998105s 2024-03-25T19:38:59.843 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.844+0000 7fa1b4923640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fa1b013f710 0x7fa1b01db1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:52300/0 (socket says 172.21.15.44:52300) 2024-03-25T19:38:59.843 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.844+0000 7fa1b4923640 1 -- 172.21.15.44:0/3788966751 learned_addr learned my addr 172.21.15.44:0/3788966751 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:38:59.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.844+0000 7fa1a5ffb640 10 client.?.objecter ms_handle_connect 0x7fa1b013f710 2024-03-25T19:38:59.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.844+0000 7fa1a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:59.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.846+0000 7fa1a5ffb640 10 client.?.objecter ms_handle_connect 0x7fa1b00a98f0 2024-03-25T19:38:59.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.846+0000 7fa1a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:59.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.851+0000 7fa1a5ffb640 10 client.?.objecter ms_handle_connect 0x7fa1b01430c0 2024-03-25T19:38:59.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.851+0000 7fa1a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:59.861 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.861+0000 7fa1a5ffb640 1 -- 172.21.15.44:0/3788966751 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640090272 0 0) 0x7fa1b00b7940 con 0x7fa1b00a98f0 2024-03-25T19:38:59.861 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.861+0000 7fa1a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:59.861 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.861+0000 7fa1a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:59.861 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.861+0000 7fa1a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:59.862 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.861+0000 7fa1a5ffb640 10 monclient(hunting): my global_id is 4378 2024-03-25T19:38:59.862 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.861+0000 7fa1a5ffb640 1 -- 172.21.15.44:0/3788966751 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa194004eb0 con 0x7fa1b00a98f0 2024-03-25T19:38:59.864 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.864+0000 7fa1a5ffb640 1 -- 172.21.15.44:0/3788966751 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335484070 0 0) 0x7fa1b0136e20 con 0x7fa1b013f710 2024-03-25T19:38:59.864 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.864+0000 7fa1a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:59.864 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.864+0000 7fa1a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:59.864 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.864+0000 7fa1a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:59.864 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.864+0000 7fa1a5ffb640 10 monclient(hunting): my global_id is 4398 2024-03-25T19:38:59.864 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.864+0000 7fa1a5ffb640 1 -- 172.21.15.44:0/3788966751 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1b00b7940 con 0x7fa1b013f710 2024-03-25T19:38:59.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.871+0000 7fa1a5ffb640 1 -- 172.21.15.44:0/3788966751 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2404364074 0 0) 0x7fa1b01351a0 con 0x7fa1b01430c0 2024-03-25T19:38:59.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.871+0000 7fa1a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:59.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.871+0000 7fa1a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:59.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.871+0000 7fa1a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:59.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.871+0000 7fa1a5ffb640 10 monclient(hunting): my global_id is 4367 2024-03-25T19:38:59.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.871+0000 7fa1a5ffb640 1 -- 172.21.15.44:0/3788966751 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1b0136e20 con 0x7fa1b01430c0 2024-03-25T19:38:59.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.874+0000 7fa1a5ffb640 1 -- 172.21.15.44:0/3788966751 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4239009226 0 0) 0x7fa1b00b7940 con 0x7fa1b013f710 2024-03-25T19:38:59.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.874+0000 7fa1a5ffb640 1 -- 172.21.15.44:0/3788966751 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1b01351a0 con 0x7fa1b013f710 2024-03-25T19:38:59.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.874+0000 7fa1a5ffb640 1 -- 172.21.15.44:0/3788966751 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fa1a00019b0 con 0x7fa1b013f710 2024-03-25T19:38:59.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.874+0000 7fa1a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:59.875 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.874+0000 7fa1a5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:38:59.875 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.874+0000 7fa1a5ffb640 10 monclient(hunting): dump: 2024-03-25T19:38:59.875 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:38:59.875 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:38:59.875 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:59.875 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:38:59.875 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:59.875 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:38:59.875 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:38:59.875 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:38:59.875 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:38:59.875 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:38:59.875 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.874+0000 7fa1a5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:59.879 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.879+0000 7fa1a5ffb640 1 -- 172.21.15.44:0/3788966751 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2852077398 0 0) 0x7fa1b01351a0 con 0x7fa1b013f710 2024-03-25T19:38:59.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.879+0000 7fa1a5ffb640 1 -- 172.21.15.44:0/3788966751 >> v1:172.21.15.44:6790/0 conn(0x7fa1b01430c0 legacy=0x7fa1b01e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:59.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.879+0000 7fa1a5ffb640 1 -- 172.21.15.44:0/3788966751 >> v1:172.21.15.111:6789/0 conn(0x7fa1b00a98f0 legacy=0x7fa1b01daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:59.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.879+0000 7fa1a5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:38:59.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.879+0000 7fa1a5ffb640 1 monclient: found mon.a 2024-03-25T19:38:59.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.879+0000 7fa1a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:59.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.879+0000 7fa1a5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:38:59.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.879+0000 7fa1a5ffb640 1 -- 172.21.15.44:0/3788966751 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b0135330 con 0x7fa1b013f710 2024-03-25T19:38:59.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.879+0000 7fa1a5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:38:59.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.879+0000 7fa1a5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:38:59.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.879+0000 7fa1a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:59.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.879+0000 7fa1b7388640 5 monclient: authenticate success, global_id 4398 2024-03-25T19:38:59.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.879+0000 7fa1b7388640 10 monclient: _renew_subs 2024-03-25T19:38:59.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.879+0000 7fa1b7388640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:38:59.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.879+0000 7fa1b7388640 1 -- 172.21.15.44:0/3788966751 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1b0135560 con 0x7fa1b013f710 2024-03-25T19:38:59.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.879+0000 7fa1b7388640 10 client.4398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:59.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.880+0000 7fa1b7388640 10 monclient: _renew_subs 2024-03-25T19:38:59.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.880+0000 7fa1b7388640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:38:59.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.880+0000 7fa1b7388640 1 -- 172.21.15.44:0/3788966751 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1b013b490 con 0x7fa1b013f710 2024-03-25T19:38:59.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.880+0000 7fa1b7388640 1 librados: init done 2024-03-25T19:38:59.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.881+0000 7fa1b7388640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:59.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.881+0000 7fa1b7388640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:59.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.881+0000 7fa1b7388640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:38:59.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:38:59.881+0000 7fa1b7388640 1 -- 172.21.15.44:0/3788966751 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1b00b7730 con 0x7fa1b013f710 2024-03-25T19:39:00.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.188+0000 7fa1a5ffb640 1 -- 172.21.15.44:0/3788966751 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1a0003610 con 0x7fa1b013f710 2024-03-25T19:39:00.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.189+0000 7fa1a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:00.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.189+0000 7fa1a5ffb640 1 -- 172.21.15.44:0/3788966751 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fa1a00039e0 con 0x7fa1b013f710 2024-03-25T19:39:00.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.189+0000 7fa1a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:00.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.189+0000 7fa1a5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:00.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.189+0000 7fa1a5ffb640 10 monclient: dump: 2024-03-25T19:39:00.189 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:00.189 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:00.189 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:00.189 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:00.189 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:00.189 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:00.190 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:00.190 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:00.190 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:00.190 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:00.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.189+0000 7fa1a77fe640 4 set_mon_vals no callback set 2024-03-25T19:39:00.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.189+0000 7fa1a5ffb640 1 -- 172.21.15.44:0/3788966751 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7fa1a0003c00 con 0x7fa1b013f710 2024-03-25T19:39:00.191 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.190+0000 7fa1b5124640 10 monclient: get_auth_request con 0x7fa19406de20 auth_method 0 2024-03-25T19:39:00.191 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.190+0000 7fa1a5ffb640 10 client.4398.objecter ms_handle_connect 0x7fa19406de20 2024-03-25T19:39:00.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.196+0000 7fa1a5ffb640 1 -- 172.21.15.44:0/3788966751 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7fa1a0054800 con 0x7fa1b013f710 2024-03-25T19:39:00.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.196+0000 7fa1a5ffb640 10 client.4398.objecter ms_dispatch 0x7fa1b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:00.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.196+0000 7fa1a5ffb640 3 client.4398.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:00.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.196+0000 7fa1a5ffb640 3 client.4398.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:00.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.196+0000 7fa1a5ffb640 20 client.4398.objecter dump_active .. 0 homeless 2024-03-25T19:39:00.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.196+0000 7fa1a5ffb640 1 -- 172.21.15.44:0/3788966751 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa1a0050310 con 0x7fa1b013f710 2024-03-25T19:39:00.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.196+0000 7fa1a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:00.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.196+0000 7fa1a5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:00.451 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.451+0000 7fa1b7388640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:00.452 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.451+0000 7fa1b7388640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:00.452 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.451+0000 7fa1b7388640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:39:00.452 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.451+0000 7fa1b7388640 1 -- 172.21.15.44:0/3788966751 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1b00627b0 con 0x7fa1b013f710 2024-03-25T19:39:00.483 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.483+0000 7fa1a5ffb640 1 -- 172.21.15.44:0/3788966751 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7fa1a0052240 con 0x7fa1b013f710 2024-03-25T19:39:00.483 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.483+0000 7fa1a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:00.483 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.483+0000 7fa1a5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:00.483 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:39:00.484 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:00.486 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.486+0000 7fa1b7388640 10 librados: watch_flush enter 2024-03-25T19:39:00.486 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.486+0000 7fa1b7388640 10 librados: watch_flush exit 2024-03-25T19:39:00.486 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.486+0000 7fa1b7388640 20 client.4398.objecter shutdown clearing up homeless session... 2024-03-25T19:39:00.486 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.486+0000 7fa1b7388640 10 client.4398.objecter successfully canceled tick 2024-03-25T19:39:00.486 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.486+0000 7fa1b7388640 1 -- 172.21.15.44:0/3788966751 >> v1:172.21.15.111:6800/3883215793 conn(0x7fa19406de20 legacy=0x7fa1940702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:00.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.486+0000 7fa1b7388640 10 monclient: shutdown 2024-03-25T19:39:00.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.486+0000 7fa1b7388640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:00.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.486+0000 7fa1b7388640 1 -- 172.21.15.44:0/3788966751 >> v1:172.21.15.44:6789/0 conn(0x7fa1b013f710 legacy=0x7fa1b01db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:00.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.486+0000 7fa1b7388640 1 -- 172.21.15.44:0/3788966751 shutdown_connections 2024-03-25T19:39:00.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.486+0000 7fa1b7388640 1 -- 172.21.15.44:0/3788966751 >> 172.21.15.44:0/3788966751 conn(0x7fa1b00b7ea0 msgr2=0x7fa1b0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:00.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.486+0000 7fa1b7388640 1 -- 172.21.15.44:0/3788966751 shutdown_connections 2024-03-25T19:39:00.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.486+0000 7fa1b7388640 1 -- 172.21.15.44:0/3788966751 wait complete. 2024-03-25T19:39:00.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:00.486+0000 7fa1b7388640 1 librados: shutdown 2024-03-25T19:39:00.498 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:07.304 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:07.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.418+0000 7f997a396640 10 monclient: get_monmap_and_config 2024-03-25T19:39:07.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.418+0000 7f997a396640 10 monclient: build_initial_monmap 2024-03-25T19:39:07.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.418+0000 7f997a396640 10 monclient: monmap: 2024-03-25T19:39:07.419 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:39:07.419 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:07.419 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:39:07.419779+0000 2024-03-25T19:39:07.419 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:39:07.419779+0000 2024-03-25T19:39:07.419 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:07.419 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:07.419 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:39:07.419 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:39:07.419 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:39:07.419 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:07.420 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.419+0000 7f997a396640 1 Processor -- start 2024-03-25T19:39:07.420 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.419+0000 7f997a396640 1 -- start start 2024-03-25T19:39:07.420 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.419+0000 7f997a396640 10 monclient: init 2024-03-25T19:39:07.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.420+0000 7f997a396640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:07.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.420+0000 7f997a396640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:39:07.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.420+0000 7f997a396640 10 monclient(hunting): picked mon.noname-c con 0x7f99740b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T19:39:07.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.420+0000 7f997a396640 10 monclient(hunting): picked mon.noname-b con 0x7f99740b1730 addr v1:172.21.15.44:6790/0 2024-03-25T19:39:07.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.420+0000 7f997a396640 10 monclient(hunting): picked mon.noname-a con 0x7f997414be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:39:07.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.420+0000 7f997a396640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99740af940 con 0x7f997414be90 2024-03-25T19:39:07.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.420+0000 7f997a396640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99740b0a70 con 0x7f99740b4d60 2024-03-25T19:39:07.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.420+0000 7f997a396640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99740b0d50 con 0x7f99740b1730 2024-03-25T19:39:07.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.420+0000 7f997a396640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:07.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.420+0000 7f997a396640 10 monclient(hunting): authenticate will time out at 5611.590992s 2024-03-25T19:39:07.423 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.423+0000 7f9978933640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f997414be90 0x7f997414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:56866/0 (socket says 172.21.15.44:56866) 2024-03-25T19:39:07.423 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.423+0000 7f9978933640 1 -- 172.21.15.44:0/1927677188 learned_addr learned my addr 172.21.15.44:0/1927677188 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:39:07.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.441+0000 7f9972ffd640 1 -- 172.21.15.44:0/1927677188 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179938865 0 0) 0x7f99740b0d50 con 0x7f99740b1730 2024-03-25T19:39:07.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.441+0000 7f9972ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:07.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.441+0000 7f9972ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:07.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.441+0000 7f9972ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:07.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.441+0000 7f9972ffd640 10 monclient(hunting): my global_id is 4370 2024-03-25T19:39:07.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.441+0000 7f9972ffd640 1 -- 172.21.15.44:0/1927677188 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f995c003520 con 0x7f99740b1730 2024-03-25T19:39:07.445 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.445+0000 7f9972ffd640 1 -- 172.21.15.44:0/1927677188 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3245929953 0 0) 0x7f995c003520 con 0x7f99740b1730 2024-03-25T19:39:07.445 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.445+0000 7f9972ffd640 1 -- 172.21.15.44:0/1927677188 >> v1:172.21.15.111:6789/0 conn(0x7f99740b4d60 legacy=0x7f997413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:07.445 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.445+0000 7f9972ffd640 1 -- 172.21.15.44:0/1927677188 >> v1:172.21.15.44:6789/0 conn(0x7f997414be90 legacy=0x7f997414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:07.445 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.445+0000 7f9972ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:39:07.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.445+0000 7f9972ffd640 1 monclient: found mon.noname-b 2024-03-25T19:39:07.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.445+0000 7f9972ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:07.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.445+0000 7f9972ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:39:07.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.445+0000 7f9972ffd640 1 -- 172.21.15.44:0/1927677188 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99740b1030 con 0x7f99740b1730 2024-03-25T19:39:07.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.445+0000 7f9972ffd640 10 monclient: _finish_auth 0 2024-03-25T19:39:07.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.445+0000 7f9972ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:39:07.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.445+0000 7f9972ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:07.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.445+0000 7f997a396640 5 monclient: authenticate success, global_id 4370 2024-03-25T19:39:07.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.445+0000 7f997a396640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:07.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.447+0000 7f9972ffd640 1 -- 172.21.15.44:0/1927677188 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f9968001690 con 0x7f99740b1730 2024-03-25T19:39:07.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.447+0000 7f9972ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:07.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.447+0000 7f9972ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:07.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.447+0000 7f9972ffd640 10 monclient: dump: 2024-03-25T19:39:07.447 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:07.447 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:07.447 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:07.447 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:07.447 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:07.447 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:07.447 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:07.448 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:07.448 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:07.448 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:07.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.447+0000 7f997a396640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:07.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.456+0000 7f9972ffd640 1 -- 172.21.15.44:0/1927677188 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9968001920 con 0x7f99740b1730 2024-03-25T19:39:07.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.456+0000 7f9972ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:07.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.456+0000 7f9972ffd640 1 -- 172.21.15.44:0/1927677188 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f9968001c30 con 0x7f99740b1730 2024-03-25T19:39:07.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.456+0000 7f997a396640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:07.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.456+0000 7f997a396640 4 set_mon_vals no callback set 2024-03-25T19:39:07.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.456+0000 7f9972ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:07.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.456+0000 7f9972ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:07.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.456+0000 7f9972ffd640 10 monclient: dump: 2024-03-25T19:39:07.456 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:07.456 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:07.456 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:07.456 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:07.456 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:07.456 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:07.457 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:07.457 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:07.457 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:07.457 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:07.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.457+0000 7f997a396640 10 monclient: shutdown 2024-03-25T19:39:07.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.457+0000 7f997a396640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:07.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.457+0000 7f997a396640 1 -- 172.21.15.44:0/1927677188 >> v1:172.21.15.44:6790/0 conn(0x7f99740b1730 legacy=0x7f99740b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:07.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.457+0000 7f997a396640 1 -- 172.21.15.44:0/1927677188 shutdown_connections 2024-03-25T19:39:07.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.457+0000 7f997a396640 1 -- 172.21.15.44:0/1927677188 >> 172.21.15.44:0/1927677188 conn(0x7f99740617a0 msgr2=0x7f9974061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:07.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.457+0000 7f997a396640 1 -- 172.21.15.44:0/1927677188 shutdown_connections 2024-03-25T19:39:07.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.457+0000 7f997a396640 1 -- 172.21.15.44:0/1927677188 wait complete. 2024-03-25T19:39:07.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.458+0000 7f997a396640 10 monclient: build_initial_monmap 2024-03-25T19:39:07.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.458+0000 7f997a396640 10 monclient: monmap: 2024-03-25T19:39:07.459 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:39:07.459 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:07.459 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:39:07.459 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:39:07.459 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:07.459 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:39:07.459 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:39:07.459 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:39:07.459 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:39:07.459 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:07.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.458+0000 7f997a396640 1 librados: starting msgr at 2024-03-25T19:39:07.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.458+0000 7f997a396640 1 librados: starting objecter 2024-03-25T19:39:07.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.458+0000 7f997a396640 1 Processor -- start 2024-03-25T19:39:07.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.458+0000 7f997a396640 1 -- start start 2024-03-25T19:39:07.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.458+0000 7f997a396640 1 librados: setting wanted keys 2024-03-25T19:39:07.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.458+0000 7f997a396640 1 librados: calling monclient init 2024-03-25T19:39:07.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.458+0000 7f997a396640 10 monclient: init 2024-03-25T19:39:07.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.459+0000 7f997a396640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:07.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.459+0000 7f997a396640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:39:07.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.459+0000 7f997a396640 10 monclient(hunting): picked mon.noname-c con 0x7f99740b1730 addr v1:172.21.15.44:6790/0 2024-03-25T19:39:07.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.459+0000 7f997a396640 10 monclient(hunting): picked mon.noname-a con 0x7f99740b4d60 addr v1:172.21.15.44:6789/0 2024-03-25T19:39:07.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.459+0000 7f997a396640 10 monclient(hunting): picked mon.noname-b con 0x7f997414be90 addr v1:172.21.15.111:6789/0 2024-03-25T19:39:07.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.459+0000 7f997a396640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9974140ce0 con 0x7f99740b4d60 2024-03-25T19:39:07.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.459+0000 7f997a396640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9974142530 con 0x7f997414be90 2024-03-25T19:39:07.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.459+0000 7f997a396640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9974135810 con 0x7f99740b1730 2024-03-25T19:39:07.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.459+0000 7f997a396640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:07.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.459+0000 7f997a396640 10 monclient(hunting): authenticate will time out at 5611.630316s 2024-03-25T19:39:07.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.466+0000 7f9978933640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f997414be90 0x7f9974135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:54276/0 (socket says 172.21.15.44:54276) 2024-03-25T19:39:07.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.466+0000 7f9978933640 1 -- 172.21.15.44:0/891414224 learned_addr learned my addr 172.21.15.44:0/891414224 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:39:07.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.466+0000 7f9970ff9640 10 client.?.objecter ms_handle_connect 0x7f997414be90 2024-03-25T19:39:07.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.466+0000 7f9970ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:07.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.480+0000 7f9970ff9640 1 -- 172.21.15.44:0/891414224 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572927408 0 0) 0x7f9974142530 con 0x7f997414be90 2024-03-25T19:39:07.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.480+0000 7f9970ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:39:07.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.480+0000 7f9970ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:07.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.480+0000 7f9970ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:07.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.480+0000 7f9970ff9640 10 monclient(hunting): my global_id is 4384 2024-03-25T19:39:07.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.480+0000 7f9970ff9640 1 -- 172.21.15.44:0/891414224 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9960004eb0 con 0x7f997414be90 2024-03-25T19:39:07.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.481+0000 7f9970ff9640 10 client.?.objecter ms_handle_connect 0x7f99740b4d60 2024-03-25T19:39:07.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.481+0000 7f9970ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:07.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.487+0000 7f9970ff9640 10 client.?.objecter ms_handle_connect 0x7f99740b1730 2024-03-25T19:39:07.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.487+0000 7f9970ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:07.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.490+0000 7f9970ff9640 1 -- 172.21.15.44:0/891414224 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74131348 0 0) 0x7f99740b1030 con 0x7f99740b4d60 2024-03-25T19:39:07.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.490+0000 7f9970ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:39:07.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.490+0000 7f9970ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:07.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.490+0000 7f9970ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:07.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.490+0000 7f9970ff9640 10 monclient(hunting): my global_id is 4404 2024-03-25T19:39:07.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.490+0000 7f9970ff9640 1 -- 172.21.15.44:0/891414224 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9974142530 con 0x7f99740b4d60 2024-03-25T19:39:07.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.493+0000 7f9970ff9640 1 -- 172.21.15.44:0/891414224 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3889736928 0 0) 0x7f9960004eb0 con 0x7f997414be90 2024-03-25T19:39:07.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.493+0000 7f9970ff9640 1 -- 172.21.15.44:0/891414224 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99740b1030 con 0x7f997414be90 2024-03-25T19:39:07.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.493+0000 7f9970ff9640 1 -- 172.21.15.44:0/891414224 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f996c001b10 con 0x7f997414be90 2024-03-25T19:39:07.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.493+0000 7f9970ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:07.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.493+0000 7f9970ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:07.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.493+0000 7f9970ff9640 10 monclient(hunting): dump: 2024-03-25T19:39:07.494 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:07.494 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:07.494 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:07.494 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:07.494 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:07.494 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:07.494 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:07.494 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:07.494 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:07.494 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:07.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.493+0000 7f9970ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:07.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.504+0000 7f9970ff9640 1 -- 172.21.15.44:0/891414224 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254984302 0 0) 0x7f9974135810 con 0x7f99740b1730 2024-03-25T19:39:07.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.504+0000 7f9970ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:39:07.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.504+0000 7f9970ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:07.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.504+0000 7f9970ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:07.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.504+0000 7f9970ff9640 10 monclient(hunting): my global_id is 4373 2024-03-25T19:39:07.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.504+0000 7f9970ff9640 1 -- 172.21.15.44:0/891414224 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9960004eb0 con 0x7f99740b1730 2024-03-25T19:39:07.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.505+0000 7f9970ff9640 1 -- 172.21.15.44:0/891414224 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3284102799 0 0) 0x7f99740b1030 con 0x7f997414be90 2024-03-25T19:39:07.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.505+0000 7f9970ff9640 1 -- 172.21.15.44:0/891414224 >> v1:172.21.15.44:6790/0 conn(0x7f99740b1730 legacy=0x7f99741429c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:07.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.505+0000 7f9970ff9640 1 -- 172.21.15.44:0/891414224 >> v1:172.21.15.44:6789/0 conn(0x7f99740b4d60 legacy=0x7f99741349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:07.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.505+0000 7f9970ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:39:07.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.505+0000 7f9970ff9640 1 monclient: found mon.b 2024-03-25T19:39:07.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.505+0000 7f9970ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:07.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.505+0000 7f9970ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:39:07.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.505+0000 7f9970ff9640 1 -- 172.21.15.44:0/891414224 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99741421e0 con 0x7f997414be90 2024-03-25T19:39:07.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.505+0000 7f9970ff9640 10 monclient: _finish_auth 0 2024-03-25T19:39:07.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.506+0000 7f9970ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:39:07.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.506+0000 7f9970ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:07.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.506+0000 7f997a396640 5 monclient: authenticate success, global_id 4384 2024-03-25T19:39:07.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.506+0000 7f997a396640 10 monclient: _renew_subs 2024-03-25T19:39:07.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.506+0000 7f997a396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:39:07.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.506+0000 7f997a396640 1 -- 172.21.15.44:0/891414224 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9974135b00 con 0x7f997414be90 2024-03-25T19:39:07.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.506+0000 7f997a396640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:07.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.506+0000 7f997a396640 10 monclient: _renew_subs 2024-03-25T19:39:07.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.506+0000 7f997a396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:39:07.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.506+0000 7f997a396640 1 -- 172.21.15.44:0/891414224 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99741430d0 con 0x7f997414be90 2024-03-25T19:39:07.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.506+0000 7f997a396640 1 librados: init done 2024-03-25T19:39:07.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.506+0000 7f997a396640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:07.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.507+0000 7f997a396640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:07.509 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.507+0000 7f997a396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:39:07.509 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.507+0000 7f997a396640 1 -- 172.21.15.44:0/891414224 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9974143380 con 0x7f997414be90 2024-03-25T19:39:07.788 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.788+0000 7f9970ff9640 1 -- 172.21.15.44:0/891414224 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f996c002000 con 0x7f997414be90 2024-03-25T19:39:07.788 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.788+0000 7f9970ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:07.788 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.788+0000 7f9970ff9640 1 -- 172.21.15.44:0/891414224 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f996c004d00 con 0x7f997414be90 2024-03-25T19:39:07.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.788+0000 7f9970ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:07.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.788+0000 7f9970ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:07.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.788+0000 7f9970ff9640 10 monclient: dump: 2024-03-25T19:39:07.789 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:07.789 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:07.789 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:07.789 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:07.789 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:07.789 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:07.790 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:07.790 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:07.790 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:07.790 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:07.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.788+0000 7f99727fc640 4 set_mon_vals no callback set 2024-03-25T19:39:07.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.789+0000 7f9970ff9640 1 -- 172.21.15.44:0/891414224 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f996c004f20 con 0x7f997414be90 2024-03-25T19:39:07.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.790+0000 7f9973fff640 10 monclient: get_auth_request con 0x7f996006de00 auth_method 0 2024-03-25T19:39:07.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.790+0000 7f9970ff9640 10 client.4384.objecter ms_handle_connect 0x7f996006de00 2024-03-25T19:39:07.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.794+0000 7f9970ff9640 1 -- 172.21.15.44:0/891414224 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f996c054980 con 0x7f997414be90 2024-03-25T19:39:07.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.794+0000 7f9970ff9640 10 client.4384.objecter ms_dispatch 0x7f9974000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:07.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.794+0000 7f9970ff9640 3 client.4384.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:07.794 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.794+0000 7f9970ff9640 3 client.4384.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:07.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.794+0000 7f9970ff9640 20 client.4384.objecter dump_active .. 0 homeless 2024-03-25T19:39:07.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.794+0000 7f9970ff9640 1 -- 172.21.15.44:0/891414224 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f996c050490 con 0x7f997414be90 2024-03-25T19:39:07.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.794+0000 7f9970ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:07.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:07.794+0000 7f9970ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:08.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:08.058+0000 7f997a396640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:08.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:08.058+0000 7f997a396640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:08.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:08.058+0000 7f997a396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:39:08.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:08.058+0000 7f997a396640 1 -- 172.21.15.44:0/891414224 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99740627b0 con 0x7f997414be90 2024-03-25T19:39:08.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:08.098+0000 7f9970ff9640 1 -- 172.21.15.44:0/891414224 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f996c0523c0 con 0x7f997414be90 2024-03-25T19:39:08.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:08.098+0000 7f9970ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:08.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:08.098+0000 7f9970ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:08.098 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:39:08.098 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:08.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:08.100+0000 7f997a396640 10 librados: watch_flush enter 2024-03-25T19:39:08.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:08.100+0000 7f997a396640 10 librados: watch_flush exit 2024-03-25T19:39:08.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:08.100+0000 7f997a396640 20 client.4384.objecter shutdown clearing up homeless session... 2024-03-25T19:39:08.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:08.100+0000 7f997a396640 10 client.4384.objecter successfully canceled tick 2024-03-25T19:39:08.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:08.101+0000 7f997a396640 1 -- 172.21.15.44:0/891414224 >> v1:172.21.15.111:6800/3883215793 conn(0x7f996006de00 legacy=0x7f99600702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:08.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:08.101+0000 7f997a396640 10 monclient: shutdown 2024-03-25T19:39:08.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:08.101+0000 7f997a396640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:08.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:08.101+0000 7f997a396640 1 -- 172.21.15.44:0/891414224 >> v1:172.21.15.111:6789/0 conn(0x7f997414be90 legacy=0x7f9974135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:08.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:08.101+0000 7f997a396640 1 -- 172.21.15.44:0/891414224 shutdown_connections 2024-03-25T19:39:08.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:08.101+0000 7f997a396640 1 -- 172.21.15.44:0/891414224 >> 172.21.15.44:0/891414224 conn(0x7f99740617a0 msgr2=0x7f997413c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:08.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:08.101+0000 7f997a396640 1 -- 172.21.15.44:0/891414224 shutdown_connections 2024-03-25T19:39:08.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:08.101+0000 7f997a396640 1 -- 172.21.15.44:0/891414224 wait complete. 2024-03-25T19:39:08.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:08.101+0000 7f997a396640 1 librados: shutdown 2024-03-25T19:39:08.112 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:14.919 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:15.037 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.037+0000 7f3959293640 10 monclient: get_monmap_and_config 2024-03-25T19:39:15.037 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.037+0000 7f3959293640 10 monclient: build_initial_monmap 2024-03-25T19:39:15.037 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.037+0000 7f3959293640 10 monclient: monmap: 2024-03-25T19:39:15.037 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:39:15.037 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:15.037 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:39:15.038421+0000 2024-03-25T19:39:15.037 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:39:15.038421+0000 2024-03-25T19:39:15.037 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:15.038 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:15.038 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:39:15.038 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:39:15.038 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:39:15.038 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:15.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.037+0000 7f3959293640 1 Processor -- start 2024-03-25T19:39:15.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.038+0000 7f3959293640 1 -- start start 2024-03-25T19:39:15.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.038+0000 7f3959293640 10 monclient: init 2024-03-25T19:39:15.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.039+0000 7f3959293640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:15.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.039+0000 7f3959293640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:39:15.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.039+0000 7f3959293640 10 monclient(hunting): picked mon.noname-b con 0x7f39540a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T19:39:15.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.039+0000 7f3959293640 10 monclient(hunting): picked mon.noname-a con 0x7f395413f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:39:15.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.039+0000 7f3959293640 10 monclient(hunting): picked mon.noname-c con 0x7f39541430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:39:15.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.039+0000 7f3959293640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3954142520 con 0x7f395413f710 2024-03-25T19:39:15.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.039+0000 7f3959293640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39540b28a0 con 0x7f39541430c0 2024-03-25T19:39:15.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.039+0000 7f3959293640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39540b2a30 con 0x7f39540a98f0 2024-03-25T19:39:15.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.039+0000 7f3959293640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:15.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.039+0000 7f3959293640 10 monclient(hunting): authenticate will time out at 5619.210426s 2024-03-25T19:39:15.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.044+0000 7f3952ffd640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f39540a98f0 0x7f39540a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:56310/0 (socket says 172.21.15.44:56310) 2024-03-25T19:39:15.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.044+0000 7f3952ffd640 1 -- 172.21.15.44:0/3500966746 learned_addr learned my addr 172.21.15.44:0/3500966746 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:39:15.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.053+0000 7f3951ffb640 1 -- 172.21.15.44:0/3500966746 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675821795 0 0) 0x7f3954142520 con 0x7f395413f710 2024-03-25T19:39:15.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.053+0000 7f3951ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:15.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.053+0000 7f3951ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:15.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.053+0000 7f3951ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:15.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.053+0000 7f3951ffb640 10 monclient(hunting): my global_id is 4407 2024-03-25T19:39:15.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.053+0000 7f3951ffb640 1 -- 172.21.15.44:0/3500966746 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f393c003520 con 0x7f395413f710 2024-03-25T19:39:15.061 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.061+0000 7f3951ffb640 1 -- 172.21.15.44:0/3500966746 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308414429 0 0) 0x7f39540b2a30 con 0x7f39540a98f0 2024-03-25T19:39:15.061 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.061+0000 7f3951ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:15.061 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.061+0000 7f3951ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:15.061 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.061+0000 7f3951ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:15.061 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.061+0000 7f3951ffb640 10 monclient(hunting): my global_id is 4376 2024-03-25T19:39:15.061 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.061+0000 7f3951ffb640 1 -- 172.21.15.44:0/3500966746 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3954142520 con 0x7f39540a98f0 2024-03-25T19:39:15.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.063+0000 7f3951ffb640 1 -- 172.21.15.44:0/3500966746 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1531067387 0 0) 0x7f393c003520 con 0x7f395413f710 2024-03-25T19:39:15.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.063+0000 7f3951ffb640 1 -- 172.21.15.44:0/3500966746 >> v1:172.21.15.44:6790/0 conn(0x7f39540a98f0 legacy=0x7f39540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:15.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.063+0000 7f3951ffb640 1 -- 172.21.15.44:0/3500966746 >> v1:172.21.15.111:6789/0 conn(0x7f39541430c0 legacy=0x7f395414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:15.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.063+0000 7f3951ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:39:15.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.063+0000 7f3951ffb640 1 monclient: found mon.noname-a 2024-03-25T19:39:15.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.063+0000 7f3951ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:15.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.063+0000 7f3951ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:39:15.064 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.063+0000 7f3951ffb640 1 -- 172.21.15.44:0/3500966746 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39540b44c0 con 0x7f395413f710 2024-03-25T19:39:15.064 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.063+0000 7f3951ffb640 10 monclient: _finish_auth 0 2024-03-25T19:39:15.064 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.063+0000 7f3951ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:39:15.064 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.063+0000 7f3951ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:15.064 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.063+0000 7f3951ffb640 1 -- 172.21.15.44:0/3500966746 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3948001690 con 0x7f395413f710 2024-03-25T19:39:15.064 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.063+0000 7f3959293640 5 monclient: authenticate success, global_id 4407 2024-03-25T19:39:15.064 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.063+0000 7f3959293640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:15.064 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.063+0000 7f3951ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:15.064 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.063+0000 7f3951ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:15.064 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.063+0000 7f3951ffb640 10 monclient: dump: 2024-03-25T19:39:15.064 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:15.064 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:15.064 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:15.064 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:15.064 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:15.065 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:15.065 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:15.065 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:15.065 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:15.065 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:15.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.063+0000 7f3959293640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:15.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.070+0000 7f3951ffb640 1 -- 172.21.15.44:0/3500966746 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39480017f0 con 0x7f395413f710 2024-03-25T19:39:15.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.070+0000 7f3951ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:15.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.070+0000 7f3951ffb640 1 -- 172.21.15.44:0/3500966746 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3948001c00 con 0x7f395413f710 2024-03-25T19:39:15.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.070+0000 7f3959293640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:15.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.070+0000 7f3959293640 4 set_mon_vals no callback set 2024-03-25T19:39:15.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.070+0000 7f3951ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:15.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.070+0000 7f3951ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:15.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.070+0000 7f3951ffb640 10 monclient: dump: 2024-03-25T19:39:15.070 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:15.070 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:15.070 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:15.071 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:15.071 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:15.071 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:15.071 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:15.071 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:15.071 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:15.071 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:15.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.071+0000 7f3959293640 10 monclient: shutdown 2024-03-25T19:39:15.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.071+0000 7f3959293640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:15.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.071+0000 7f3959293640 1 -- 172.21.15.44:0/3500966746 >> v1:172.21.15.44:6789/0 conn(0x7f395413f710 legacy=0x7f39541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:15.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.071+0000 7f3959293640 1 -- 172.21.15.44:0/3500966746 shutdown_connections 2024-03-25T19:39:15.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.071+0000 7f3959293640 1 -- 172.21.15.44:0/3500966746 >> 172.21.15.44:0/3500966746 conn(0x7f39540b7ea0 msgr2=0x7f39540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:15.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.071+0000 7f3959293640 1 -- 172.21.15.44:0/3500966746 shutdown_connections 2024-03-25T19:39:15.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.071+0000 7f3959293640 1 -- 172.21.15.44:0/3500966746 wait complete. 2024-03-25T19:39:15.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.072+0000 7f3959293640 10 monclient: build_initial_monmap 2024-03-25T19:39:15.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.072+0000 7f3959293640 10 monclient: monmap: 2024-03-25T19:39:15.072 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:39:15.072 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:15.072 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:39:15.073 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:39:15.073 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:15.073 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:39:15.073 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:39:15.073 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:39:15.073 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:39:15.073 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:15.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.072+0000 7f3959293640 1 librados: starting msgr at 2024-03-25T19:39:15.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.072+0000 7f3959293640 1 librados: starting objecter 2024-03-25T19:39:15.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.073+0000 7f3959293640 1 Processor -- start 2024-03-25T19:39:15.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.073+0000 7f3959293640 1 -- start start 2024-03-25T19:39:15.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.073+0000 7f3959293640 1 librados: setting wanted keys 2024-03-25T19:39:15.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.073+0000 7f3959293640 1 librados: calling monclient init 2024-03-25T19:39:15.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.073+0000 7f3959293640 10 monclient: init 2024-03-25T19:39:15.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.074+0000 7f3959293640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:15.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.074+0000 7f3959293640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:39:15.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.074+0000 7f3959293640 10 monclient(hunting): picked mon.noname-a con 0x7f39540a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:39:15.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.074+0000 7f3959293640 10 monclient(hunting): picked mon.noname-b con 0x7f395413f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:39:15.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.074+0000 7f3959293640 10 monclient(hunting): picked mon.noname-c con 0x7f39541430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:39:15.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.074+0000 7f3959293640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39541d82d0 con 0x7f39540a98f0 2024-03-25T19:39:15.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.074+0000 7f3959293640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39540abb90 con 0x7f395413f710 2024-03-25T19:39:15.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.074+0000 7f3959293640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39541d76a0 con 0x7f39541430c0 2024-03-25T19:39:15.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.074+0000 7f3959293640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:15.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.074+0000 7f3959293640 10 monclient(hunting): authenticate will time out at 5619.245037s 2024-03-25T19:39:15.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.083+0000 7f39527fc640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f395413f710 0x7f39541d7bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:40878/0 (socket says 172.21.15.44:40878) 2024-03-25T19:39:15.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.083+0000 7f39527fc640 1 -- 172.21.15.44:0/2881253291 learned_addr learned my addr 172.21.15.44:0/2881253291 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:39:15.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.084+0000 7f393b7fe640 10 client.?.objecter ms_handle_connect 0x7f395413f710 2024-03-25T19:39:15.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.084+0000 7f393b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:15.088 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.088+0000 7f393b7fe640 1 -- 172.21.15.44:0/2881253291 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578372202 0 0) 0x7f39540abb90 con 0x7f395413f710 2024-03-25T19:39:15.088 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.088+0000 7f393b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:15.088 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.088+0000 7f393b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:15.089 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.088+0000 7f393b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:15.089 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.088+0000 7f393b7fe640 10 monclient(hunting): my global_id is 4390 2024-03-25T19:39:15.089 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.088+0000 7f393b7fe640 1 -- 172.21.15.44:0/2881253291 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3940004eb0 con 0x7f395413f710 2024-03-25T19:39:15.090 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.091+0000 7f393b7fe640 10 client.?.objecter ms_handle_connect 0x7f39540a98f0 2024-03-25T19:39:15.090 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.091+0000 7f393b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:15.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.097+0000 7f393b7fe640 1 -- 172.21.15.44:0/2881253291 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 316551332 0 0) 0x7f3940004eb0 con 0x7f395413f710 2024-03-25T19:39:15.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.097+0000 7f393b7fe640 1 -- 172.21.15.44:0/2881253291 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39540abb90 con 0x7f395413f710 2024-03-25T19:39:15.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.097+0000 7f393b7fe640 1 -- 172.21.15.44:0/2881253291 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3948001bc0 con 0x7f395413f710 2024-03-25T19:39:15.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.097+0000 7f393b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:15.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.097+0000 7f393b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:15.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.097+0000 7f393b7fe640 10 monclient(hunting): dump: 2024-03-25T19:39:15.098 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:15.098 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:15.098 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:15.098 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:15.098 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:15.098 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:15.098 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:15.098 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:15.098 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:15.098 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:15.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.097+0000 7f393b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:15.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.098+0000 7f393b7fe640 10 client.?.objecter ms_handle_connect 0x7f39541430c0 2024-03-25T19:39:15.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.098+0000 7f393b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:15.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.099+0000 7f393b7fe640 1 -- 172.21.15.44:0/2881253291 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592912206 0 0) 0x7f39541d82d0 con 0x7f39540a98f0 2024-03-25T19:39:15.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.099+0000 7f393b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:15.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.099+0000 7f393b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:15.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.099+0000 7f393b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:15.100 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.099+0000 7f393b7fe640 10 monclient(hunting): my global_id is 4410 2024-03-25T19:39:15.100 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.099+0000 7f393b7fe640 1 -- 172.21.15.44:0/2881253291 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3940004eb0 con 0x7f39540a98f0 2024-03-25T19:39:15.108 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.108+0000 7f393b7fe640 1 -- 172.21.15.44:0/2881253291 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332274395 0 0) 0x7f39541d76a0 con 0x7f39541430c0 2024-03-25T19:39:15.108 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.108+0000 7f393b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:15.108 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.108+0000 7f393b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:15.108 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.108+0000 7f393b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:15.108 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.108+0000 7f393b7fe640 10 monclient(hunting): my global_id is 4379 2024-03-25T19:39:15.108 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.108+0000 7f393b7fe640 1 -- 172.21.15.44:0/2881253291 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39541d82d0 con 0x7f39541430c0 2024-03-25T19:39:15.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.111+0000 7f393b7fe640 1 -- 172.21.15.44:0/2881253291 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2177552143 0 0) 0x7f3940004eb0 con 0x7f39540a98f0 2024-03-25T19:39:15.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.111+0000 7f393b7fe640 1 -- 172.21.15.44:0/2881253291 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39541d76a0 con 0x7f39540a98f0 2024-03-25T19:39:15.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.111+0000 7f393b7fe640 1 -- 172.21.15.44:0/2881253291 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3944001930 con 0x7f39540a98f0 2024-03-25T19:39:15.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.111+0000 7f393b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:15.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.111+0000 7f393b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:15.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.111+0000 7f393b7fe640 10 monclient(hunting): dump: 2024-03-25T19:39:15.111 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:15.111 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:15.111 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:15.111 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:15.112 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:15.112 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:15.112 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:15.112 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:15.112 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:15.112 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:15.118 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.117+0000 7f393b7fe640 1 -- 172.21.15.44:0/2881253291 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3922619886 0 0) 0x7f39540abb90 con 0x7f395413f710 2024-03-25T19:39:15.118 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.117+0000 7f393b7fe640 1 -- 172.21.15.44:0/2881253291 >> v1:172.21.15.44:6790/0 conn(0x7f39541430c0 legacy=0x7f3954134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:15.118 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.117+0000 7f393b7fe640 1 -- 172.21.15.44:0/2881253291 >> v1:172.21.15.44:6789/0 conn(0x7f39540a98f0 legacy=0x7f39541385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:15.118 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.117+0000 7f393b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:39:15.118 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.117+0000 7f393b7fe640 1 monclient: found mon.b 2024-03-25T19:39:15.118 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.117+0000 7f393b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:15.118 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.117+0000 7f393b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:39:15.118 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.117+0000 7f393b7fe640 1 -- 172.21.15.44:0/2881253291 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3954134720 con 0x7f395413f710 2024-03-25T19:39:15.118 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.117+0000 7f393b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:39:15.118 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.117+0000 7f393b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:39:15.118 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.117+0000 7f393b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:15.118 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.117+0000 7f3959293640 5 monclient: authenticate success, global_id 4390 2024-03-25T19:39:15.119 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.117+0000 7f3959293640 10 monclient: _renew_subs 2024-03-25T19:39:15.119 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.117+0000 7f3959293640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:39:15.119 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.117+0000 7f3959293640 1 -- 172.21.15.44:0/2881253291 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3954134970 con 0x7f395413f710 2024-03-25T19:39:15.119 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.118+0000 7f3959293640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:15.119 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.118+0000 7f3959293640 10 monclient: _renew_subs 2024-03-25T19:39:15.119 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.118+0000 7f3959293640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:39:15.119 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.118+0000 7f3959293640 1 -- 172.21.15.44:0/2881253291 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3954137a10 con 0x7f395413f710 2024-03-25T19:39:15.119 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.118+0000 7f3959293640 1 librados: init done 2024-03-25T19:39:15.119 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.119+0000 7f3959293640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:15.120 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.119+0000 7f3959293640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:15.120 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.119+0000 7f3959293640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:39:15.120 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.119+0000 7f3959293640 1 -- 172.21.15.44:0/2881253291 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39541d7830 con 0x7f395413f710 2024-03-25T19:39:15.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.140+0000 7f393b7fe640 1 -- 172.21.15.44:0/2881253291 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39480038a0 con 0x7f395413f710 2024-03-25T19:39:15.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.140+0000 7f393b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:15.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.140+0000 7f393b7fe640 1 -- 172.21.15.44:0/2881253291 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3948003cb0 con 0x7f395413f710 2024-03-25T19:39:15.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.140+0000 7f393b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:15.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.140+0000 7f393b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:15.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.140+0000 7f393b7fe640 10 monclient: dump: 2024-03-25T19:39:15.141 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:15.141 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:15.141 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:15.141 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:15.141 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:15.141 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:15.141 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:15.141 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:15.141 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:15.141 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:15.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.140+0000 7f39517fa640 4 set_mon_vals no callback set 2024-03-25T19:39:15.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.142+0000 7f393b7fe640 1 -- 172.21.15.44:0/2881253291 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f3948003ed0 con 0x7f395413f710 2024-03-25T19:39:15.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.143+0000 7f3952ffd640 10 monclient: get_auth_request con 0x7f394006dfd0 auth_method 0 2024-03-25T19:39:15.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.143+0000 7f393b7fe640 10 client.4390.objecter ms_handle_connect 0x7f394006dfd0 2024-03-25T19:39:15.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.145+0000 7f393b7fe640 1 -- 172.21.15.44:0/2881253291 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f3948054920 con 0x7f395413f710 2024-03-25T19:39:15.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.145+0000 7f393b7fe640 10 client.4390.objecter ms_dispatch 0x7f3954000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:15.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.145+0000 7f393b7fe640 3 client.4390.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:15.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.145+0000 7f393b7fe640 3 client.4390.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:15.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.145+0000 7f393b7fe640 20 client.4390.objecter dump_active .. 0 homeless 2024-03-25T19:39:15.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.416+0000 7f393b7fe640 1 -- 172.21.15.44:0/2881253291 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3948050430 con 0x7f395413f710 2024-03-25T19:39:15.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.416+0000 7f393b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:15.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.416+0000 7f393b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:15.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.635+0000 7f3959293640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:15.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.635+0000 7f3959293640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:15.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.635+0000 7f3959293640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:39:15.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.635+0000 7f3959293640 1 -- 172.21.15.44:0/2881253291 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3954074cc0 con 0x7f395413f710 2024-03-25T19:39:15.661 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.661+0000 7f393b7fe640 1 -- 172.21.15.44:0/2881253291 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f3948052360 con 0x7f395413f710 2024-03-25T19:39:15.661 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.661+0000 7f393b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:15.661 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.661+0000 7f393b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:15.661 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:39:15.662 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:15.664 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.664+0000 7f3959293640 10 librados: watch_flush enter 2024-03-25T19:39:15.664 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.664+0000 7f3959293640 10 librados: watch_flush exit 2024-03-25T19:39:15.664 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.664+0000 7f3959293640 20 client.4390.objecter shutdown clearing up homeless session... 2024-03-25T19:39:15.664 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.664+0000 7f3959293640 10 client.4390.objecter successfully canceled tick 2024-03-25T19:39:15.664 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.664+0000 7f3959293640 1 -- 172.21.15.44:0/2881253291 >> v1:172.21.15.111:6800/3883215793 conn(0x7f394006dfd0 legacy=0x7f3940070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:15.665 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.664+0000 7f3959293640 10 monclient: shutdown 2024-03-25T19:39:15.665 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.664+0000 7f3959293640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:15.665 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.664+0000 7f3959293640 1 -- 172.21.15.44:0/2881253291 >> v1:172.21.15.111:6789/0 conn(0x7f395413f710 legacy=0x7f39541d7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:15.665 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.664+0000 7f3959293640 1 -- 172.21.15.44:0/2881253291 shutdown_connections 2024-03-25T19:39:15.665 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.664+0000 7f3959293640 1 -- 172.21.15.44:0/2881253291 >> 172.21.15.44:0/2881253291 conn(0x7f39540b7ea0 msgr2=0x7f3954074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:15.665 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.664+0000 7f3959293640 1 -- 172.21.15.44:0/2881253291 shutdown_connections 2024-03-25T19:39:15.665 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.664+0000 7f3959293640 1 -- 172.21.15.44:0/2881253291 wait complete. 2024-03-25T19:39:15.665 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:15.664+0000 7f3959293640 1 librados: shutdown 2024-03-25T19:39:15.675 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:19.231 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:04:10.133 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:19.281 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:04:10.183 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:19.283 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:04:10.185 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:19.286 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:04:10.188 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:19.301 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:04:10.203 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:19.320 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:04:10.221 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:19.343 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:04:10.223 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:19.354 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:04:10.234 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:19.384 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:04:10.264 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:22.482 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:22.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.596+0000 7f8ce89a0640 10 monclient: get_monmap_and_config 2024-03-25T19:39:22.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.596+0000 7f8ce89a0640 10 monclient: build_initial_monmap 2024-03-25T19:39:22.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.596+0000 7f8ce89a0640 10 monclient: monmap: 2024-03-25T19:39:22.597 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:39:22.597 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:22.597 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:39:22.597811+0000 2024-03-25T19:39:22.597 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:39:22.597811+0000 2024-03-25T19:39:22.597 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:22.597 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:22.597 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:39:22.597 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:39:22.597 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:39:22.597 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:22.598 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.597+0000 7f8ce89a0640 1 Processor -- start 2024-03-25T19:39:22.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.597+0000 7f8ce89a0640 1 -- start start 2024-03-25T19:39:22.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.597+0000 7f8ce89a0640 10 monclient: init 2024-03-25T19:39:22.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.598+0000 7f8ce89a0640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:22.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.598+0000 7f8ce89a0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:39:22.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.598+0000 7f8ce89a0640 10 monclient(hunting): picked mon.noname-a con 0x7f8ce40b48e0 addr v1:172.21.15.44:6789/0 2024-03-25T19:39:22.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.598+0000 7f8ce89a0640 10 monclient(hunting): picked mon.noname-c con 0x7f8ce413fc60 addr v1:172.21.15.111:6789/0 2024-03-25T19:39:22.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.598+0000 7f8ce89a0640 10 monclient(hunting): picked mon.noname-b con 0x7f8ce40b3f90 addr v1:172.21.15.44:6790/0 2024-03-25T19:39:22.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.598+0000 7f8ce89a0640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce40a9d10 con 0x7f8ce40b48e0 2024-03-25T19:39:22.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.598+0000 7f8ce89a0640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce40ab000 con 0x7f8ce413fc60 2024-03-25T19:39:22.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.598+0000 7f8ce89a0640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce40a9680 con 0x7f8ce40b3f90 2024-03-25T19:39:22.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.598+0000 7f8ce89a0640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:22.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.598+0000 7f8ce89a0640 10 monclient(hunting): authenticate will time out at 5626.769749s 2024-03-25T19:39:22.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.603+0000 7f8ce2ffd640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f8ce413fc60 0x7f8ce4145770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:52494/0 (socket says 172.21.15.44:52494) 2024-03-25T19:39:22.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.603+0000 7f8ce2ffd640 1 -- 172.21.15.44:0/3993687174 learned_addr learned my addr 172.21.15.44:0/3993687174 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:39:22.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.607+0000 7f8ce27fc640 1 -- 172.21.15.44:0/3993687174 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4022724838 0 0) 0x7f8ce40ab000 con 0x7f8ce413fc60 2024-03-25T19:39:22.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.607+0000 7f8ce27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:22.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.607+0000 7f8ce27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:22.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.607+0000 7f8ce27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:22.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.607+0000 7f8ce27fc640 10 monclient(hunting): my global_id is 4393 2024-03-25T19:39:22.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.607+0000 7f8ce27fc640 1 -- 172.21.15.44:0/3993687174 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ccc003520 con 0x7f8ce413fc60 2024-03-25T19:39:22.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.623+0000 7f8ce27fc640 1 -- 172.21.15.44:0/3993687174 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1435390349 0 0) 0x7f8ccc003520 con 0x7f8ce413fc60 2024-03-25T19:39:22.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.623+0000 7f8ce27fc640 1 -- 172.21.15.44:0/3993687174 >> v1:172.21.15.44:6790/0 conn(0x7f8ce40b3f90 legacy=0x7f8ce414dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:22.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.623+0000 7f8ce27fc640 1 -- 172.21.15.44:0/3993687174 >> v1:172.21.15.44:6789/0 conn(0x7f8ce40b48e0 legacy=0x7f8ce413f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:22.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.623+0000 7f8ce27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:39:22.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.623+0000 7f8ce27fc640 1 monclient: found mon.noname-c 2024-03-25T19:39:22.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.623+0000 7f8ce27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:22.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.623+0000 7f8ce27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:39:22.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.623+0000 7f8ce27fc640 1 -- 172.21.15.44:0/3993687174 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce40a9810 con 0x7f8ce413fc60 2024-03-25T19:39:22.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.623+0000 7f8ce27fc640 10 monclient: _finish_auth 0 2024-03-25T19:39:22.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.623+0000 7f8ce27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:39:22.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.623+0000 7f8ce27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:22.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.623+0000 7f8ce27fc640 1 -- 172.21.15.44:0/3993687174 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f8cd8001690 con 0x7f8ce413fc60 2024-03-25T19:39:22.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.623+0000 7f8ce89a0640 5 monclient: authenticate success, global_id 4393 2024-03-25T19:39:22.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.623+0000 7f8ce89a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:22.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.623+0000 7f8ce27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:22.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.623+0000 7f8ce27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:22.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.623+0000 7f8ce27fc640 10 monclient: dump: 2024-03-25T19:39:22.624 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:22.624 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:22.624 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:22.624 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:22.625 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:22.625 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:22.625 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:22.625 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:22.625 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:22.625 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:22.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.623+0000 7f8ce89a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:22.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.635+0000 7f8ce27fc640 1 -- 172.21.15.44:0/3993687174 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cd80017f0 con 0x7f8ce413fc60 2024-03-25T19:39:22.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.635+0000 7f8ce27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:22.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.635+0000 7f8ce27fc640 1 -- 172.21.15.44:0/3993687174 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f8cd8001c40 con 0x7f8ce413fc60 2024-03-25T19:39:22.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.635+0000 7f8ce27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:22.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.635+0000 7f8ce27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:22.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.635+0000 7f8ce27fc640 10 monclient: dump: 2024-03-25T19:39:22.636 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:22.636 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:22.636 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:22.636 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:22.636 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:22.636 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:22.636 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:22.636 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:22.636 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:22.637 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:22.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.635+0000 7f8ce89a0640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:22.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.635+0000 7f8ce89a0640 4 set_mon_vals no callback set 2024-03-25T19:39:22.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.636+0000 7f8ce89a0640 10 monclient: shutdown 2024-03-25T19:39:22.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.636+0000 7f8ce89a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:22.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.636+0000 7f8ce89a0640 1 -- 172.21.15.44:0/3993687174 >> v1:172.21.15.111:6789/0 conn(0x7f8ce413fc60 legacy=0x7f8ce4145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:22.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.636+0000 7f8ce89a0640 1 -- 172.21.15.44:0/3993687174 shutdown_connections 2024-03-25T19:39:22.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.636+0000 7f8ce89a0640 1 -- 172.21.15.44:0/3993687174 >> 172.21.15.44:0/3993687174 conn(0x7f8ce40615c0 msgr2=0x7f8ce40619d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:22.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.636+0000 7f8ce89a0640 1 -- 172.21.15.44:0/3993687174 shutdown_connections 2024-03-25T19:39:22.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.636+0000 7f8ce89a0640 1 -- 172.21.15.44:0/3993687174 wait complete. 2024-03-25T19:39:22.639 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.637+0000 7f8ce89a0640 10 monclient: build_initial_monmap 2024-03-25T19:39:22.639 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.637+0000 7f8ce89a0640 10 monclient: monmap: 2024-03-25T19:39:22.639 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:39:22.639 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:22.639 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:39:22.639 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:39:22.639 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:22.639 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:39:22.639 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:39:22.639 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:39:22.639 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:39:22.639 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:22.639 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.638+0000 7f8ce89a0640 1 librados: starting msgr at 2024-03-25T19:39:22.639 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.638+0000 7f8ce89a0640 1 librados: starting objecter 2024-03-25T19:39:22.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.638+0000 7f8ce89a0640 1 Processor -- start 2024-03-25T19:39:22.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.638+0000 7f8ce89a0640 1 -- start start 2024-03-25T19:39:22.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.638+0000 7f8ce89a0640 1 librados: setting wanted keys 2024-03-25T19:39:22.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.638+0000 7f8ce89a0640 1 librados: calling monclient init 2024-03-25T19:39:22.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.638+0000 7f8ce89a0640 10 monclient: init 2024-03-25T19:39:22.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.639+0000 7f8ce89a0640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:22.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.639+0000 7f8ce89a0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:39:22.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.639+0000 7f8ce89a0640 10 monclient(hunting): picked mon.noname-a con 0x7f8ce40b3f90 addr v1:172.21.15.44:6789/0 2024-03-25T19:39:22.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.639+0000 7f8ce89a0640 10 monclient(hunting): picked mon.noname-c con 0x7f8ce40b48e0 addr v1:172.21.15.44:6790/0 2024-03-25T19:39:22.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.639+0000 7f8ce89a0640 10 monclient(hunting): picked mon.noname-b con 0x7f8ce413fc60 addr v1:172.21.15.111:6789/0 2024-03-25T19:39:22.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.639+0000 7f8ce89a0640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce40ad1e0 con 0x7f8ce40b3f90 2024-03-25T19:39:22.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.639+0000 7f8ce89a0640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce40ad450 con 0x7f8ce413fc60 2024-03-25T19:39:22.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.639+0000 7f8ce89a0640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce4140410 con 0x7f8ce40b48e0 2024-03-25T19:39:22.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.639+0000 7f8ce89a0640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:22.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.639+0000 7f8ce89a0640 10 monclient(hunting): authenticate will time out at 5626.810247s 2024-03-25T19:39:22.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.646+0000 7f8ce3fff640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f8ce413fc60 0x7f8ce41e28a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:52540/0 (socket says 172.21.15.44:52540) 2024-03-25T19:39:22.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.646+0000 7f8ce3fff640 1 -- 172.21.15.44:0/741378116 learned_addr learned my addr 172.21.15.44:0/741378116 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:39:22.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.646+0000 7f8ccbfff640 10 client.?.objecter ms_handle_connect 0x7f8ce413fc60 2024-03-25T19:39:22.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.646+0000 7f8ccbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:22.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.656+0000 7f8ccbfff640 1 -- 172.21.15.44:0/741378116 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56382970 0 0) 0x7f8ce40ad450 con 0x7f8ce413fc60 2024-03-25T19:39:22.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.656+0000 7f8ccbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:39:22.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.656+0000 7f8ccbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:22.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.656+0000 7f8ccbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:22.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.656+0000 7f8ccbfff640 10 monclient(hunting): my global_id is 4396 2024-03-25T19:39:22.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.656+0000 7f8ccbfff640 1 -- 172.21.15.44:0/741378116 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cd0004eb0 con 0x7f8ce413fc60 2024-03-25T19:39:22.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.663+0000 7f8ccbfff640 10 client.?.objecter ms_handle_connect 0x7f8ce40b48e0 2024-03-25T19:39:22.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.663+0000 7f8ccbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:22.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.666+0000 7f8ccbfff640 10 client.?.objecter ms_handle_connect 0x7f8ce40b3f90 2024-03-25T19:39:22.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.666+0000 7f8ccbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:22.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.678+0000 7f8ccbfff640 1 -- 172.21.15.44:0/741378116 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3841456040 0 0) 0x7f8cd0004eb0 con 0x7f8ce413fc60 2024-03-25T19:39:22.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.678+0000 7f8ccbfff640 1 -- 172.21.15.44:0/741378116 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ce40ad450 con 0x7f8ce413fc60 2024-03-25T19:39:22.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.678+0000 7f8ccbfff640 1 -- 172.21.15.44:0/741378116 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f8cdc001cd0 con 0x7f8ce413fc60 2024-03-25T19:39:22.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.678+0000 7f8ccbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:22.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.678+0000 7f8ccbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:22.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.678+0000 7f8ccbfff640 10 monclient(hunting): dump: 2024-03-25T19:39:22.679 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:22.679 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:22.679 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:22.679 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:22.679 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:22.679 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:22.679 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:22.679 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:22.679 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:22.680 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:22.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.678+0000 7f8ccbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:22.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.679+0000 7f8ccbfff640 1 -- 172.21.15.44:0/741378116 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190381161 0 0) 0x7f8ce4140410 con 0x7f8ce40b48e0 2024-03-25T19:39:22.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.679+0000 7f8ccbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:39:22.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.679+0000 7f8ccbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:22.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.679+0000 7f8ccbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:22.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.679+0000 7f8ccbfff640 10 monclient(hunting): my global_id is 4385 2024-03-25T19:39:22.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.679+0000 7f8ccbfff640 1 -- 172.21.15.44:0/741378116 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cd0004eb0 con 0x7f8ce40b48e0 2024-03-25T19:39:22.685 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.685+0000 7f8ccbfff640 1 -- 172.21.15.44:0/741378116 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1983806543 0 0) 0x7f8ce40ad1e0 con 0x7f8ce40b3f90 2024-03-25T19:39:22.685 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.685+0000 7f8ccbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:39:22.685 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.685+0000 7f8ccbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:22.685 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.685+0000 7f8ccbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:22.685 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.685+0000 7f8ccbfff640 10 monclient(hunting): my global_id is 4416 2024-03-25T19:39:22.685 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.685+0000 7f8ccbfff640 1 -- 172.21.15.44:0/741378116 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ce4140410 con 0x7f8ce40b3f90 2024-03-25T19:39:22.697 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.697+0000 7f8ccbfff640 1 -- 172.21.15.44:0/741378116 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4265258910 0 0) 0x7f8ce40ad450 con 0x7f8ce413fc60 2024-03-25T19:39:22.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.697+0000 7f8ccbfff640 1 -- 172.21.15.44:0/741378116 >> v1:172.21.15.44:6790/0 conn(0x7f8ce40b48e0 legacy=0x7f8ce4137eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:22.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.697+0000 7f8ccbfff640 1 -- 172.21.15.44:0/741378116 >> v1:172.21.15.44:6789/0 conn(0x7f8ce40b3f90 legacy=0x7f8ce41dab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:22.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.697+0000 7f8ccbfff640 10 monclient: _finish_hunting 0 2024-03-25T19:39:22.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.697+0000 7f8ccbfff640 1 monclient: found mon.b 2024-03-25T19:39:22.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.697+0000 7f8ccbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:22.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.697+0000 7f8ccbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:39:22.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.697+0000 7f8ccbfff640 1 -- 172.21.15.44:0/741378116 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce41405a0 con 0x7f8ce413fc60 2024-03-25T19:39:22.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.697+0000 7f8ccbfff640 10 monclient: _finish_auth 0 2024-03-25T19:39:22.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.697+0000 7f8ccbfff640 10 monclient: _check_auth_tickets 2024-03-25T19:39:22.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.697+0000 7f8ccbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:22.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.697+0000 7f8ce89a0640 5 monclient: authenticate success, global_id 4396 2024-03-25T19:39:22.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.697+0000 7f8ce89a0640 10 monclient: _renew_subs 2024-03-25T19:39:22.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.697+0000 7f8ce89a0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:39:22.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.697+0000 7f8ce89a0640 1 -- 172.21.15.44:0/741378116 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ce41db260 con 0x7f8ce413fc60 2024-03-25T19:39:22.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.697+0000 7f8ce89a0640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:22.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.697+0000 7f8ce89a0640 10 monclient: _renew_subs 2024-03-25T19:39:22.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.697+0000 7f8ce89a0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:39:22.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.697+0000 7f8ce89a0640 1 -- 172.21.15.44:0/741378116 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce41387a0 con 0x7f8ce413fc60 2024-03-25T19:39:22.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.697+0000 7f8ce89a0640 1 librados: init done 2024-03-25T19:39:22.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.698+0000 7f8ce89a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:22.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.698+0000 7f8ce89a0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:22.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.698+0000 7f8ce89a0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:39:22.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.698+0000 7f8ce89a0640 1 -- 172.21.15.44:0/741378116 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ce40aa180 con 0x7f8ce413fc60 2024-03-25T19:39:22.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.978+0000 7f8ccbfff640 1 -- 172.21.15.44:0/741378116 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cdc003990 con 0x7f8ce413fc60 2024-03-25T19:39:22.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.978+0000 7f8ccbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:22.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.978+0000 7f8ccbfff640 1 -- 172.21.15.44:0/741378116 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f8cdc003c00 con 0x7f8ce413fc60 2024-03-25T19:39:22.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.978+0000 7f8ccbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:22.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.978+0000 7f8ccbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:22.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.978+0000 7f8ce1ffb640 4 set_mon_vals no callback set 2024-03-25T19:39:22.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.978+0000 7f8ccbfff640 10 monclient: dump: 2024-03-25T19:39:22.979 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:22.979 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:22.979 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:22.979 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:22.979 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:22.979 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:22.979 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:22.980 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:22.980 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:22.980 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:22.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.978+0000 7f8ccbfff640 1 -- 172.21.15.44:0/741378116 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f8cdc003e20 con 0x7f8ce413fc60 2024-03-25T19:39:22.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.979+0000 7f8ce37fe640 10 monclient: get_auth_request con 0x7f8cd006de60 auth_method 0 2024-03-25T19:39:22.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.979+0000 7f8ccbfff640 10 client.4396.objecter ms_handle_connect 0x7f8cd006de60 2024-03-25T19:39:22.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.989+0000 7f8ccbfff640 1 -- 172.21.15.44:0/741378116 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f8cdc054980 con 0x7f8ce413fc60 2024-03-25T19:39:22.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.989+0000 7f8ccbfff640 10 client.4396.objecter ms_dispatch 0x7f8ce4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:22.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.989+0000 7f8ccbfff640 3 client.4396.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:22.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.989+0000 7f8ccbfff640 3 client.4396.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:22.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.989+0000 7f8ccbfff640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-25T19:39:22.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.989+0000 7f8ccbfff640 1 -- 172.21.15.44:0/741378116 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8cdc050490 con 0x7f8ce413fc60 2024-03-25T19:39:22.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.989+0000 7f8ccbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:22.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:22.989+0000 7f8ccbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:23.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:23.248+0000 7f8ce89a0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:23.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:23.248+0000 7f8ce89a0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:23.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:23.248+0000 7f8ce89a0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:39:23.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:23.248+0000 7f8ce89a0640 1 -- 172.21.15.44:0/741378116 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ce40627b0 con 0x7f8ce413fc60 2024-03-25T19:39:23.282 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:23.282+0000 7f8ccbfff640 1 -- 172.21.15.44:0/741378116 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f8cdc0523c0 con 0x7f8ce413fc60 2024-03-25T19:39:23.282 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:23.282+0000 7f8ccbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:23.282 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:23.282+0000 7f8ccbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:23.282 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:39:23.282 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:23.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:23.284+0000 7f8ce89a0640 10 librados: watch_flush enter 2024-03-25T19:39:23.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:23.284+0000 7f8ce89a0640 10 librados: watch_flush exit 2024-03-25T19:39:23.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:23.284+0000 7f8ce89a0640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-25T19:39:23.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:23.284+0000 7f8ce89a0640 10 client.4396.objecter successfully canceled tick 2024-03-25T19:39:23.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:23.284+0000 7f8ce89a0640 1 -- 172.21.15.44:0/741378116 >> v1:172.21.15.111:6800/3883215793 conn(0x7f8cd006de60 legacy=0x7f8cd0070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:23.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:23.284+0000 7f8ce89a0640 10 monclient: shutdown 2024-03-25T19:39:23.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:23.284+0000 7f8ce89a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:23.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:23.284+0000 7f8ce89a0640 1 -- 172.21.15.44:0/741378116 >> v1:172.21.15.111:6789/0 conn(0x7f8ce413fc60 legacy=0x7f8ce41e28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:23.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:23.284+0000 7f8ce89a0640 1 -- 172.21.15.44:0/741378116 shutdown_connections 2024-03-25T19:39:23.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:23.284+0000 7f8ce89a0640 1 -- 172.21.15.44:0/741378116 >> 172.21.15.44:0/741378116 conn(0x7f8ce40615c0 msgr2=0x7f8ce413eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:23.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:23.284+0000 7f8ce89a0640 1 -- 172.21.15.44:0/741378116 shutdown_connections 2024-03-25T19:39:23.286 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:23.284+0000 7f8ce89a0640 1 -- 172.21.15.44:0/741378116 wait complete. 2024-03-25T19:39:23.286 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:23.284+0000 7f8ce89a0640 1 librados: shutdown 2024-03-25T19:39:23.297 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:30.103 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:30.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.216+0000 7ff766ac9640 10 monclient: get_monmap_and_config 2024-03-25T19:39:30.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.216+0000 7ff766ac9640 10 monclient: build_initial_monmap 2024-03-25T19:39:30.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.216+0000 7ff766ac9640 10 monclient: monmap: 2024-03-25T19:39:30.216 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:39:30.216 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:30.216 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:39:30.217260+0000 2024-03-25T19:39:30.216 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:39:30.217260+0000 2024-03-25T19:39:30.217 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:30.217 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:30.217 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:39:30.217 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:39:30.217 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:39:30.217 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:30.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.216+0000 7ff766ac9640 1 Processor -- start 2024-03-25T19:39:30.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.217+0000 7ff766ac9640 1 -- start start 2024-03-25T19:39:30.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.217+0000 7ff766ac9640 10 monclient: init 2024-03-25T19:39:30.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.217+0000 7ff766ac9640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:30.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.217+0000 7ff766ac9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:39:30.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.217+0000 7ff766ac9640 10 monclient(hunting): picked mon.noname-b con 0x7ff7600b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T19:39:30.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.217+0000 7ff766ac9640 10 monclient(hunting): picked mon.noname-a con 0x7ff7600b1730 addr v1:172.21.15.44:6789/0 2024-03-25T19:39:30.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.217+0000 7ff766ac9640 10 monclient(hunting): picked mon.noname-c con 0x7ff76014be90 addr v1:172.21.15.111:6789/0 2024-03-25T19:39:30.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.218+0000 7ff766ac9640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7600af940 con 0x7ff7600b1730 2024-03-25T19:39:30.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.218+0000 7ff766ac9640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7600b0a70 con 0x7ff76014be90 2024-03-25T19:39:30.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.218+0000 7ff766ac9640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7600b0d50 con 0x7ff7600b4d60 2024-03-25T19:39:30.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.218+0000 7ff766ac9640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:30.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.218+0000 7ff766ac9640 10 monclient(hunting): authenticate will time out at 5634.388876s 2024-03-25T19:39:30.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.231+0000 7ff765066640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff76014be90 0x7ff76014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:47902/0 (socket says 172.21.15.44:47902) 2024-03-25T19:39:30.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.231+0000 7ff765066640 1 -- 172.21.15.44:0/2780054589 learned_addr learned my addr 172.21.15.44:0/2780054589 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:39:30.242 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.242+0000 7ff75f7fe640 1 -- 172.21.15.44:0/2780054589 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864996679 0 0) 0x7ff7600b0a70 con 0x7ff76014be90 2024-03-25T19:39:30.242 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.242+0000 7ff75f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:30.242 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.242+0000 7ff75f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:30.242 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.242+0000 7ff75f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:30.242 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.242+0000 7ff75f7fe640 10 monclient(hunting): my global_id is 4399 2024-03-25T19:39:30.243 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.242+0000 7ff75f7fe640 1 -- 172.21.15.44:0/2780054589 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff750003520 con 0x7ff76014be90 2024-03-25T19:39:30.254 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.254+0000 7ff75f7fe640 1 -- 172.21.15.44:0/2780054589 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427291504 0 0) 0x7ff7600b0d50 con 0x7ff7600b4d60 2024-03-25T19:39:30.254 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.254+0000 7ff75f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:30.254 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.254+0000 7ff75f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:30.254 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.254+0000 7ff75f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:30.254 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.254+0000 7ff75f7fe640 10 monclient(hunting): my global_id is 4388 2024-03-25T19:39:30.255 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.254+0000 7ff75f7fe640 1 -- 172.21.15.44:0/2780054589 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7600b0a70 con 0x7ff7600b4d60 2024-03-25T19:39:30.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.261+0000 7ff75f7fe640 1 -- 172.21.15.44:0/2780054589 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3302042149 0 0) 0x7ff750003520 con 0x7ff76014be90 2024-03-25T19:39:30.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.261+0000 7ff75f7fe640 1 -- 172.21.15.44:0/2780054589 >> v1:172.21.15.44:6790/0 conn(0x7ff7600b4d60 legacy=0x7ff76013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:30.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.261+0000 7ff75f7fe640 1 -- 172.21.15.44:0/2780054589 >> v1:172.21.15.44:6789/0 conn(0x7ff7600b1730 legacy=0x7ff7600b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:30.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.261+0000 7ff75f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:39:30.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.261+0000 7ff75f7fe640 1 monclient: found mon.noname-c 2024-03-25T19:39:30.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.261+0000 7ff75f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:30.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.261+0000 7ff75f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:39:30.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.261+0000 7ff75f7fe640 1 -- 172.21.15.44:0/2780054589 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7600b1030 con 0x7ff76014be90 2024-03-25T19:39:30.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.261+0000 7ff75f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:39:30.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.261+0000 7ff75f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:39:30.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.261+0000 7ff75f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:30.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.262+0000 7ff75f7fe640 1 -- 172.21.15.44:0/2780054589 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff758001a10 con 0x7ff76014be90 2024-03-25T19:39:30.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.262+0000 7ff75f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:30.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.262+0000 7ff75f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:30.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.262+0000 7ff75f7fe640 10 monclient: dump: 2024-03-25T19:39:30.263 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:30.263 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:30.263 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:30.263 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:30.263 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:30.263 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:30.263 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:30.263 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:30.263 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:30.264 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:30.264 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.262+0000 7ff766ac9640 5 monclient: authenticate success, global_id 4399 2024-03-25T19:39:30.264 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.262+0000 7ff766ac9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:30.264 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.265+0000 7ff75f7fe640 1 -- 172.21.15.44:0/2780054589 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff758001b70 con 0x7ff76014be90 2024-03-25T19:39:30.264 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.265+0000 7ff75f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:30.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.265+0000 7ff766ac9640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:30.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.265+0000 7ff766ac9640 4 set_mon_vals no callback set 2024-03-25T19:39:30.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.266+0000 7ff766ac9640 10 monclient: shutdown 2024-03-25T19:39:30.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.266+0000 7ff766ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:30.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.266+0000 7ff766ac9640 1 -- 172.21.15.44:0/2780054589 >> v1:172.21.15.111:6789/0 conn(0x7ff76014be90 legacy=0x7ff76014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:30.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.266+0000 7ff766ac9640 1 -- 172.21.15.44:0/2780054589 shutdown_connections 2024-03-25T19:39:30.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.266+0000 7ff766ac9640 1 -- 172.21.15.44:0/2780054589 >> 172.21.15.44:0/2780054589 conn(0x7ff7600617a0 msgr2=0x7ff760061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:30.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.266+0000 7ff766ac9640 1 -- 172.21.15.44:0/2780054589 shutdown_connections 2024-03-25T19:39:30.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.266+0000 7ff766ac9640 1 -- 172.21.15.44:0/2780054589 wait complete. 2024-03-25T19:39:30.267 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.266+0000 7ff766ac9640 10 monclient: build_initial_monmap 2024-03-25T19:39:30.267 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.266+0000 7ff766ac9640 10 monclient: monmap: 2024-03-25T19:39:30.267 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:39:30.267 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:30.267 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:39:30.267 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:39:30.267 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:30.268 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:39:30.268 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:39:30.268 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:39:30.268 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:39:30.268 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:30.268 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.267+0000 7ff766ac9640 1 librados: starting msgr at 2024-03-25T19:39:30.268 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.267+0000 7ff766ac9640 1 librados: starting objecter 2024-03-25T19:39:30.268 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.267+0000 7ff766ac9640 1 Processor -- start 2024-03-25T19:39:30.268 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.267+0000 7ff766ac9640 1 -- start start 2024-03-25T19:39:30.268 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.267+0000 7ff766ac9640 1 librados: setting wanted keys 2024-03-25T19:39:30.268 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.267+0000 7ff766ac9640 1 librados: calling monclient init 2024-03-25T19:39:30.268 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.267+0000 7ff766ac9640 10 monclient: init 2024-03-25T19:39:30.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.268+0000 7ff766ac9640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:30.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.268+0000 7ff766ac9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:39:30.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.268+0000 7ff766ac9640 10 monclient(hunting): picked mon.noname-b con 0x7ff7600b1730 addr v1:172.21.15.111:6789/0 2024-03-25T19:39:30.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.268+0000 7ff766ac9640 10 monclient(hunting): picked mon.noname-c con 0x7ff7600b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T19:39:30.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.268+0000 7ff766ac9640 10 monclient(hunting): picked mon.noname-a con 0x7ff76014be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:39:30.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.268+0000 7ff766ac9640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7600aa7d0 con 0x7ff76014be90 2024-03-25T19:39:30.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.268+0000 7ff766ac9640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7601dbe40 con 0x7ff7600b1730 2024-03-25T19:39:30.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.268+0000 7ff766ac9640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff760145740 con 0x7ff7600b4d60 2024-03-25T19:39:30.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.268+0000 7ff766ac9640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:30.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.268+0000 7ff766ac9640 10 monclient(hunting): authenticate will time out at 5634.439415s 2024-03-25T19:39:30.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.271+0000 7ff75ffff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7ff7600b4d60 0x7ff7601db020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:54094/0 (socket says 172.21.15.44:54094) 2024-03-25T19:39:30.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.271+0000 7ff75ffff640 1 -- 172.21.15.44:0/4144460982 learned_addr learned my addr 172.21.15.44:0/4144460982 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:39:30.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.272+0000 7ff75d7fa640 10 client.?.objecter ms_handle_connect 0x7ff7600b4d60 2024-03-25T19:39:30.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.272+0000 7ff75d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:30.272 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.272+0000 7ff75d7fa640 10 client.?.objecter ms_handle_connect 0x7ff7600b1730 2024-03-25T19:39:30.272 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.272+0000 7ff75d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:30.277 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.277+0000 7ff75d7fa640 1 -- 172.21.15.44:0/4144460982 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28942802 0 0) 0x7ff7601dbe40 con 0x7ff7600b1730 2024-03-25T19:39:30.277 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.277+0000 7ff75d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:39:30.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.277+0000 7ff75d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:30.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.277+0000 7ff75d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:30.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.277+0000 7ff75d7fa640 10 monclient(hunting): my global_id is 4402 2024-03-25T19:39:30.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.277+0000 7ff75d7fa640 1 -- 172.21.15.44:0/4144460982 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff744004eb0 con 0x7ff7600b1730 2024-03-25T19:39:30.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.278+0000 7ff75d7fa640 10 client.?.objecter ms_handle_connect 0x7ff76014be90 2024-03-25T19:39:30.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.278+0000 7ff75d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:30.284 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.283+0000 7ff75d7fa640 1 -- 172.21.15.44:0/4144460982 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1117980483 0 0) 0x7ff744004eb0 con 0x7ff7600b1730 2024-03-25T19:39:30.284 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.283+0000 7ff75d7fa640 1 -- 172.21.15.44:0/4144460982 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7601dbe40 con 0x7ff7600b1730 2024-03-25T19:39:30.284 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.283+0000 7ff75d7fa640 1 -- 172.21.15.44:0/4144460982 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff754001930 con 0x7ff7600b1730 2024-03-25T19:39:30.284 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.283+0000 7ff75d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:30.284 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.284+0000 7ff75d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:30.284 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.284+0000 7ff75d7fa640 10 monclient(hunting): dump: 2024-03-25T19:39:30.284 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:30.284 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:30.284 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:30.284 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:30.284 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:30.284 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:30.285 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:30.285 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:30.285 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:30.285 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:30.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.284+0000 7ff75d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:30.288 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.288+0000 7ff75d7fa640 1 -- 172.21.15.44:0/4144460982 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3297448853 0 0) 0x7ff760145740 con 0x7ff7600b4d60 2024-03-25T19:39:30.288 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.288+0000 7ff75d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:39:30.288 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.288+0000 7ff75d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:30.288 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.288+0000 7ff75d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:30.288 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.288+0000 7ff75d7fa640 10 monclient(hunting): my global_id is 4391 2024-03-25T19:39:30.288 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.288+0000 7ff75d7fa640 1 -- 172.21.15.44:0/4144460982 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff744004eb0 con 0x7ff7600b4d60 2024-03-25T19:39:30.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.290+0000 7ff75d7fa640 1 -- 172.21.15.44:0/4144460982 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2192227113 0 0) 0x7ff7601dbe40 con 0x7ff7600b1730 2024-03-25T19:39:30.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.290+0000 7ff75d7fa640 1 -- 172.21.15.44:0/4144460982 >> v1:172.21.15.44:6790/0 conn(0x7ff7600b4d60 legacy=0x7ff7601db020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:30.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.290+0000 7ff75d7fa640 1 -- 172.21.15.44:0/4144460982 >> v1:172.21.15.44:6789/0 conn(0x7ff76014be90 legacy=0x7ff7601db730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:30.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.290+0000 7ff75d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:39:30.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.290+0000 7ff75d7fa640 1 monclient: found mon.b 2024-03-25T19:39:30.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.290+0000 7ff75d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:30.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.290+0000 7ff75d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:39:30.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.290+0000 7ff75d7fa640 1 -- 172.21.15.44:0/4144460982 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7600a9470 con 0x7ff7600b1730 2024-03-25T19:39:30.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.290+0000 7ff75d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:39:30.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.290+0000 7ff75d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:39:30.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.290+0000 7ff75d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:30.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.290+0000 7ff766ac9640 5 monclient: authenticate success, global_id 4402 2024-03-25T19:39:30.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.290+0000 7ff766ac9640 10 monclient: _renew_subs 2024-03-25T19:39:30.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.290+0000 7ff766ac9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:39:30.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.290+0000 7ff766ac9640 1 -- 172.21.15.44:0/4144460982 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7600a96c0 con 0x7ff7600b1730 2024-03-25T19:39:30.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.290+0000 7ff766ac9640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:30.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.290+0000 7ff766ac9640 10 monclient: _renew_subs 2024-03-25T19:39:30.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.290+0000 7ff766ac9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:39:30.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.290+0000 7ff766ac9640 1 -- 172.21.15.44:0/4144460982 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7601434b0 con 0x7ff7600b1730 2024-03-25T19:39:30.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.290+0000 7ff766ac9640 1 librados: init done 2024-03-25T19:39:30.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.291+0000 7ff766ac9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:30.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.291+0000 7ff766ac9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:30.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.291+0000 7ff766ac9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:39:30.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.291+0000 7ff766ac9640 1 -- 172.21.15.44:0/4144460982 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7600627b0 con 0x7ff7600b1730 2024-03-25T19:39:30.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.572+0000 7ff75d7fa640 1 -- 172.21.15.44:0/4144460982 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff754003610 con 0x7ff7600b1730 2024-03-25T19:39:30.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.572+0000 7ff75d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:30.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.572+0000 7ff75d7fa640 1 -- 172.21.15.44:0/4144460982 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff754003a60 con 0x7ff7600b1730 2024-03-25T19:39:30.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.572+0000 7ff75d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:30.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.572+0000 7ff75d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:30.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.572+0000 7ff75d7fa640 10 monclient: dump: 2024-03-25T19:39:30.573 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:30.573 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:30.573 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:30.573 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:30.574 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:30.574 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:30.574 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:30.574 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:30.574 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:30.574 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:30.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.572+0000 7ff75effd640 4 set_mon_vals no callback set 2024-03-25T19:39:30.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.573+0000 7ff75d7fa640 1 -- 172.21.15.44:0/4144460982 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7ff754003c80 con 0x7ff7600b1730 2024-03-25T19:39:30.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.574+0000 7ff75ffff640 10 monclient: get_auth_request con 0x7ff74406de20 auth_method 0 2024-03-25T19:39:30.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.574+0000 7ff75d7fa640 10 client.4402.objecter ms_handle_connect 0x7ff74406de20 2024-03-25T19:39:30.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.576+0000 7ff75d7fa640 1 -- 172.21.15.44:0/4144460982 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7ff7540535f0 con 0x7ff7600b1730 2024-03-25T19:39:30.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.576+0000 7ff75d7fa640 10 client.4402.objecter ms_dispatch 0x7ff760000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:30.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.576+0000 7ff75d7fa640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:30.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.576+0000 7ff75d7fa640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:30.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.576+0000 7ff75d7fa640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-25T19:39:30.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.578+0000 7ff75d7fa640 1 -- 172.21.15.44:0/4144460982 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff75404f100 con 0x7ff7600b1730 2024-03-25T19:39:30.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.578+0000 7ff75d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:30.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.578+0000 7ff75d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:30.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.820+0000 7ff766ac9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:30.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.820+0000 7ff766ac9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:30.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.820+0000 7ff766ac9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:39:30.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.820+0000 7ff766ac9640 1 -- 172.21.15.44:0/4144460982 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff760069f40 con 0x7ff7600b1730 2024-03-25T19:39:30.843 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.843+0000 7ff75d7fa640 1 -- 172.21.15.44:0/4144460982 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7ff754051030 con 0x7ff7600b1730 2024-03-25T19:39:30.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.843+0000 7ff75d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:30.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.843+0000 7ff75d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:30.844 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:39:30.844 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:30.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.847+0000 7ff766ac9640 10 librados: watch_flush enter 2024-03-25T19:39:30.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.847+0000 7ff766ac9640 10 librados: watch_flush exit 2024-03-25T19:39:30.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.847+0000 7ff766ac9640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-25T19:39:30.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.847+0000 7ff766ac9640 10 client.4402.objecter successfully canceled tick 2024-03-25T19:39:30.847 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.847+0000 7ff766ac9640 1 -- 172.21.15.44:0/4144460982 >> v1:172.21.15.111:6800/3883215793 conn(0x7ff74406de20 legacy=0x7ff7440702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:30.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.847+0000 7ff766ac9640 10 monclient: shutdown 2024-03-25T19:39:30.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.847+0000 7ff766ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:30.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.847+0000 7ff766ac9640 1 -- 172.21.15.44:0/4144460982 >> v1:172.21.15.111:6789/0 conn(0x7ff7600b1730 legacy=0x7ff7600aabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:30.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.847+0000 7ff766ac9640 1 -- 172.21.15.44:0/4144460982 shutdown_connections 2024-03-25T19:39:30.848 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.847+0000 7ff766ac9640 1 -- 172.21.15.44:0/4144460982 >> 172.21.15.44:0/4144460982 conn(0x7ff7600617a0 msgr2=0x7ff76013c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:30.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.847+0000 7ff766ac9640 1 -- 172.21.15.44:0/4144460982 shutdown_connections 2024-03-25T19:39:30.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.847+0000 7ff766ac9640 1 -- 172.21.15.44:0/4144460982 wait complete. 2024-03-25T19:39:30.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:30.847+0000 7ff766ac9640 1 librados: shutdown 2024-03-25T19:39:30.859 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:37.665 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:37.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.781+0000 7f6a4b674640 10 monclient: get_monmap_and_config 2024-03-25T19:39:37.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.781+0000 7f6a4b674640 10 monclient: build_initial_monmap 2024-03-25T19:39:37.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.781+0000 7f6a4b674640 10 monclient: monmap: 2024-03-25T19:39:37.780 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:39:37.780 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:37.780 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:39:37.781498+0000 2024-03-25T19:39:37.780 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:39:37.781498+0000 2024-03-25T19:39:37.780 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:37.780 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:37.780 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:39:37.780 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:39:37.781 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:39:37.781 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:37.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.782+0000 7f6a4b674640 1 Processor -- start 2024-03-25T19:39:37.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.782+0000 7f6a4b674640 1 -- start start 2024-03-25T19:39:37.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.782+0000 7f6a4b674640 10 monclient: init 2024-03-25T19:39:37.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.782+0000 7f6a4b674640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:37.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.782+0000 7f6a4b674640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:39:37.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.783+0000 7f6a4b674640 10 monclient(hunting): picked mon.noname-c con 0x7f6a440a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:39:37.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.783+0000 7f6a4b674640 10 monclient(hunting): picked mon.noname-a con 0x7f6a4413f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:39:37.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.783+0000 7f6a4b674640 10 monclient(hunting): picked mon.noname-b con 0x7f6a441430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:39:37.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.783+0000 7f6a4b674640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a44142520 con 0x7f6a4413f710 2024-03-25T19:39:37.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.783+0000 7f6a4b674640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a440b28a0 con 0x7f6a440a98f0 2024-03-25T19:39:37.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.783+0000 7f6a4b674640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a440b2a30 con 0x7f6a441430c0 2024-03-25T19:39:37.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.783+0000 7f6a4b674640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:37.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.783+0000 7f6a4b674640 10 monclient(hunting): authenticate will time out at 5641.953039s 2024-03-25T19:39:37.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.790+0000 7f6a49c11640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f6a441430c0 0x7f6a4414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:58810/0 (socket says 172.21.15.44:58810) 2024-03-25T19:39:37.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.790+0000 7f6a49c11640 1 -- 172.21.15.44:0/58621602 learned_addr learned my addr 172.21.15.44:0/58621602 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:39:37.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.804+0000 7f6a3bfff640 1 -- 172.21.15.44:0/58621602 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2441856623 0 0) 0x7f6a440b28a0 con 0x7f6a440a98f0 2024-03-25T19:39:37.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.804+0000 7f6a3bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:39:37.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.804+0000 7f6a3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:37.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.804+0000 7f6a3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:37.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.804+0000 7f6a3bfff640 10 monclient(hunting): my global_id is 4405 2024-03-25T19:39:37.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.804+0000 7f6a3bfff640 1 -- 172.21.15.44:0/58621602 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a30003520 con 0x7f6a440a98f0 2024-03-25T19:39:37.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.809+0000 7f6a3bfff640 1 -- 172.21.15.44:0/58621602 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1942042579 0 0) 0x7f6a30003520 con 0x7f6a440a98f0 2024-03-25T19:39:37.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.809+0000 7f6a3bfff640 1 -- 172.21.15.44:0/58621602 >> v1:172.21.15.44:6790/0 conn(0x7f6a441430c0 legacy=0x7f6a4414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:37.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.809+0000 7f6a3bfff640 1 -- 172.21.15.44:0/58621602 >> v1:172.21.15.44:6789/0 conn(0x7f6a4413f710 legacy=0x7f6a441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:37.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.809+0000 7f6a3bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:39:37.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.809+0000 7f6a3bfff640 1 monclient: found mon.noname-c 2024-03-25T19:39:37.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.809+0000 7f6a3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:37.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.809+0000 7f6a3bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:39:37.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.809+0000 7f6a3bfff640 1 -- 172.21.15.44:0/58621602 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a440b44c0 con 0x7f6a440a98f0 2024-03-25T19:39:37.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.809+0000 7f6a3bfff640 10 monclient: _finish_auth 0 2024-03-25T19:39:37.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.809+0000 7f6a3bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:39:37.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.809+0000 7f6a3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:37.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.809+0000 7f6a3bfff640 1 -- 172.21.15.44:0/58621602 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6a40001690 con 0x7f6a440a98f0 2024-03-25T19:39:37.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.809+0000 7f6a4b674640 5 monclient: authenticate success, global_id 4405 2024-03-25T19:39:37.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.809+0000 7f6a4b674640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:37.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.809+0000 7f6a3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:37.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.809+0000 7f6a3bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:37.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.809+0000 7f6a3bfff640 10 monclient: dump: 2024-03-25T19:39:37.809 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:37.809 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:37.810 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:37.810 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:37.810 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:37.810 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:37.810 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:37.810 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:37.810 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:37.810 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:37.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.809+0000 7f6a4b674640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:37.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.817+0000 7f6a3bfff640 1 -- 172.21.15.44:0/58621602 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a400017f0 con 0x7f6a440a98f0 2024-03-25T19:39:37.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.817+0000 7f6a3bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:37.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.817+0000 7f6a3bfff640 1 -- 172.21.15.44:0/58621602 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6a40001c40 con 0x7f6a440a98f0 2024-03-25T19:39:37.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.817+0000 7f6a4b674640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:37.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.817+0000 7f6a4b674640 4 set_mon_vals no callback set 2024-03-25T19:39:37.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.817+0000 7f6a3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:37.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.817+0000 7f6a3bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:37.817 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.817+0000 7f6a3bfff640 10 monclient: dump: 2024-03-25T19:39:37.817 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:37.817 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:37.817 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:37.817 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:37.817 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:37.817 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:37.817 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:37.817 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:37.817 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:37.817 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:37.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.818+0000 7f6a4b674640 10 monclient: shutdown 2024-03-25T19:39:37.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.818+0000 7f6a4b674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:37.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.818+0000 7f6a4b674640 1 -- 172.21.15.44:0/58621602 >> v1:172.21.15.111:6789/0 conn(0x7f6a440a98f0 legacy=0x7f6a440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:37.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.818+0000 7f6a4b674640 1 -- 172.21.15.44:0/58621602 shutdown_connections 2024-03-25T19:39:37.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.818+0000 7f6a4b674640 1 -- 172.21.15.44:0/58621602 >> 172.21.15.44:0/58621602 conn(0x7f6a440b7ea0 msgr2=0x7f6a440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:37.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.818+0000 7f6a4b674640 1 -- 172.21.15.44:0/58621602 shutdown_connections 2024-03-25T19:39:37.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.818+0000 7f6a4b674640 1 -- 172.21.15.44:0/58621602 wait complete. 2024-03-25T19:39:37.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.819+0000 7f6a4b674640 10 monclient: build_initial_monmap 2024-03-25T19:39:37.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.819+0000 7f6a4b674640 10 monclient: monmap: 2024-03-25T19:39:37.819 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:39:37.819 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:37.819 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:39:37.819 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:39:37.819 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:37.819 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:39:37.819 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:39:37.819 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:39:37.819 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:39:37.819 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:37.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.820+0000 7f6a4b674640 1 librados: starting msgr at 2024-03-25T19:39:37.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.820+0000 7f6a4b674640 1 librados: starting objecter 2024-03-25T19:39:37.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.820+0000 7f6a4b674640 1 Processor -- start 2024-03-25T19:39:37.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.820+0000 7f6a4b674640 1 -- start start 2024-03-25T19:39:37.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.820+0000 7f6a4b674640 1 librados: setting wanted keys 2024-03-25T19:39:37.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.820+0000 7f6a4b674640 1 librados: calling monclient init 2024-03-25T19:39:37.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.820+0000 7f6a4b674640 10 monclient: init 2024-03-25T19:39:37.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.820+0000 7f6a4b674640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:37.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.820+0000 7f6a4b674640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:39:37.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.820+0000 7f6a4b674640 10 monclient(hunting): picked mon.noname-b con 0x7f6a440a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:39:37.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.821+0000 7f6a4b674640 10 monclient(hunting): picked mon.noname-a con 0x7f6a4413f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:39:37.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.821+0000 7f6a4b674640 10 monclient(hunting): picked mon.noname-c con 0x7f6a441430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:39:37.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.821+0000 7f6a4b674640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a441df080 con 0x7f6a4413f710 2024-03-25T19:39:37.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.821+0000 7f6a4b674640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a44070420 con 0x7f6a440a98f0 2024-03-25T19:39:37.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.821+0000 7f6a4b674640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a4406f3c0 con 0x7f6a441430c0 2024-03-25T19:39:37.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.821+0000 7f6a4b674640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:37.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.821+0000 7f6a4b674640 10 monclient(hunting): authenticate will time out at 5641.991135s 2024-03-25T19:39:37.831 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.832+0000 7f6a49c11640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f6a441430c0 0x7f6a441e64d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:58844/0 (socket says 172.21.15.44:58844) 2024-03-25T19:39:37.831 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.832+0000 7f6a49c11640 1 -- 172.21.15.44:0/3578461762 learned_addr learned my addr 172.21.15.44:0/3578461762 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:39:37.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.834+0000 7f6a39ffb640 10 client.?.objecter ms_handle_connect 0x7f6a441430c0 2024-03-25T19:39:37.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.834+0000 7f6a39ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:37.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.840+0000 7f6a39ffb640 1 -- 172.21.15.44:0/3578461762 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745074449 0 0) 0x7f6a4406f3c0 con 0x7f6a441430c0 2024-03-25T19:39:37.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.840+0000 7f6a39ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:37.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.840+0000 7f6a39ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:37.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.840+0000 7f6a39ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:37.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.840+0000 7f6a39ffb640 10 monclient(hunting): my global_id is 4397 2024-03-25T19:39:37.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.840+0000 7f6a39ffb640 1 -- 172.21.15.44:0/3578461762 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a28004eb0 con 0x7f6a441430c0 2024-03-25T19:39:37.845 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.846+0000 7f6a39ffb640 10 client.?.objecter ms_handle_connect 0x7f6a4413f710 2024-03-25T19:39:37.845 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.846+0000 7f6a39ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:37.845 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.846+0000 7f6a39ffb640 10 client.?.objecter ms_handle_connect 0x7f6a440a98f0 2024-03-25T19:39:37.845 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.846+0000 7f6a39ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:37.860 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.861+0000 7f6a39ffb640 1 -- 172.21.15.44:0/3578461762 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211412822 0 0) 0x7f6a44070420 con 0x7f6a440a98f0 2024-03-25T19:39:37.860 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.861+0000 7f6a39ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:37.860 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.861+0000 7f6a39ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:37.860 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.861+0000 7f6a39ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:37.860 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.861+0000 7f6a39ffb640 10 monclient(hunting): my global_id is 4408 2024-03-25T19:39:37.860 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.861+0000 7f6a39ffb640 1 -- 172.21.15.44:0/3578461762 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a4406f3c0 con 0x7f6a440a98f0 2024-03-25T19:39:37.861 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.862+0000 7f6a39ffb640 1 -- 172.21.15.44:0/3578461762 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1261731478 0 0) 0x7f6a28004eb0 con 0x7f6a441430c0 2024-03-25T19:39:37.861 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.862+0000 7f6a39ffb640 1 -- 172.21.15.44:0/3578461762 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a44070420 con 0x7f6a441430c0 2024-03-25T19:39:37.861 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.862+0000 7f6a39ffb640 1 -- 172.21.15.44:0/3578461762 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6a3c001b10 con 0x7f6a441430c0 2024-03-25T19:39:37.861 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.862+0000 7f6a39ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:37.861 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.862+0000 7f6a39ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:37.861 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.862+0000 7f6a39ffb640 10 monclient(hunting): dump: 2024-03-25T19:39:37.861 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:37.861 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:37.861 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:37.861 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:37.861 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:37.862 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:37.862 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:37.862 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:37.862 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:37.862 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:37.862 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.862+0000 7f6a39ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:37.865 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.866+0000 7f6a39ffb640 1 -- 172.21.15.44:0/3578461762 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 678950693 0 0) 0x7f6a441df080 con 0x7f6a4413f710 2024-03-25T19:39:37.865 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.866+0000 7f6a39ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:37.865 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.866+0000 7f6a39ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:37.865 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.866+0000 7f6a39ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:37.865 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.866+0000 7f6a39ffb640 10 monclient(hunting): my global_id is 4428 2024-03-25T19:39:37.865 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.866+0000 7f6a39ffb640 1 -- 172.21.15.44:0/3578461762 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a28004eb0 con 0x7f6a4413f710 2024-03-25T19:39:37.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.882+0000 7f6a39ffb640 1 -- 172.21.15.44:0/3578461762 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2603454171 0 0) 0x7f6a28004eb0 con 0x7f6a4413f710 2024-03-25T19:39:37.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.882+0000 7f6a39ffb640 1 -- 172.21.15.44:0/3578461762 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a441df080 con 0x7f6a4413f710 2024-03-25T19:39:37.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.882+0000 7f6a39ffb640 1 -- 172.21.15.44:0/3578461762 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6a34001830 con 0x7f6a4413f710 2024-03-25T19:39:37.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.882+0000 7f6a39ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:37.881 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.882+0000 7f6a39ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:37.882 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.882+0000 7f6a39ffb640 10 monclient(hunting): dump: 2024-03-25T19:39:37.882 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:37.882 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:37.882 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:37.882 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:37.882 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:37.882 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:37.882 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:37.882 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:37.882 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:37.882 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:37.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.885+0000 7f6a39ffb640 1 -- 172.21.15.44:0/3578461762 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 584868262 0 0) 0x7f6a4406f3c0 con 0x7f6a440a98f0 2024-03-25T19:39:37.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.885+0000 7f6a39ffb640 1 -- 172.21.15.44:0/3578461762 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a28004eb0 con 0x7f6a440a98f0 2024-03-25T19:39:37.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.885+0000 7f6a39ffb640 1 -- 172.21.15.44:0/3578461762 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6a40001b40 con 0x7f6a440a98f0 2024-03-25T19:39:37.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.885+0000 7f6a39ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:37.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.885+0000 7f6a39ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:37.884 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.885+0000 7f6a39ffb640 10 monclient(hunting): dump: 2024-03-25T19:39:37.884 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:37.884 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:37.884 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:37.884 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:37.884 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:37.884 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:37.884 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:37.885 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:37.885 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:37.885 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:37.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.887+0000 7f6a39ffb640 1 -- 172.21.15.44:0/3578461762 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3830621690 0 0) 0x7f6a44070420 con 0x7f6a441430c0 2024-03-25T19:39:37.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.887+0000 7f6a39ffb640 1 -- 172.21.15.44:0/3578461762 >> v1:172.21.15.111:6789/0 conn(0x7f6a440a98f0 legacy=0x7f6a4406ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:37.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.887+0000 7f6a39ffb640 1 -- 172.21.15.44:0/3578461762 >> v1:172.21.15.44:6789/0 conn(0x7f6a4413f710 legacy=0x7f6a441df510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:37.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.887+0000 7f6a39ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:39:37.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.887+0000 7f6a39ffb640 1 monclient: found mon.c 2024-03-25T19:39:37.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.887+0000 7f6a39ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:37.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.887+0000 7f6a39ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:39:37.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.887+0000 7f6a39ffb640 1 -- 172.21.15.44:0/3578461762 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4406f550 con 0x7f6a441430c0 2024-03-25T19:39:37.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.887+0000 7f6a39ffb640 10 monclient: _finish_auth 0 2024-03-25T19:39:37.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.887+0000 7f6a39ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:39:37.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.887+0000 7f6a39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:37.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.887+0000 7f6a4b674640 5 monclient: authenticate success, global_id 4397 2024-03-25T19:39:37.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.887+0000 7f6a4b674640 10 monclient: _renew_subs 2024-03-25T19:39:37.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.887+0000 7f6a4b674640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:39:37.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.887+0000 7f6a4b674640 1 -- 172.21.15.44:0/3578461762 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a4406e0a0 con 0x7f6a441430c0 2024-03-25T19:39:37.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.887+0000 7f6a4b674640 10 client.4397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:37.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.887+0000 7f6a4b674640 10 monclient: _renew_subs 2024-03-25T19:39:37.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.887+0000 7f6a4b674640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:39:37.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.887+0000 7f6a4b674640 1 -- 172.21.15.44:0/3578461762 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a441dfe00 con 0x7f6a441430c0 2024-03-25T19:39:37.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.887+0000 7f6a4b674640 1 librados: init done 2024-03-25T19:39:37.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.888+0000 7f6a4b674640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:37.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.888+0000 7f6a4b674640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:37.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.888+0000 7f6a4b674640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:39:37.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:37.888+0000 7f6a4b674640 1 -- 172.21.15.44:0/3578461762 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a441df230 con 0x7f6a441430c0 2024-03-25T19:39:38.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.213+0000 7f6a39ffb640 1 -- 172.21.15.44:0/3578461762 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a3c002080 con 0x7f6a441430c0 2024-03-25T19:39:38.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.213+0000 7f6a39ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:38.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.213+0000 7f6a39ffb640 1 -- 172.21.15.44:0/3578461762 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6a3c004b60 con 0x7f6a441430c0 2024-03-25T19:39:38.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.213+0000 7f6a39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:38.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.213+0000 7f6a39ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:38.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.213+0000 7f6a39ffb640 10 monclient: dump: 2024-03-25T19:39:38.213 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:38.213 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:38.213 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:38.213 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:38.213 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:38.213 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:38.213 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:38.213 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:38.213 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:38.213 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:38.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.213+0000 7f6a3b7fe640 4 set_mon_vals no callback set 2024-03-25T19:39:38.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.214+0000 7f6a39ffb640 1 -- 172.21.15.44:0/3578461762 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f6a3c01abf0 con 0x7f6a441430c0 2024-03-25T19:39:38.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.214+0000 7f6a49410640 10 monclient: get_auth_request con 0x7f6a2806e110 auth_method 0 2024-03-25T19:39:38.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.215+0000 7f6a39ffb640 10 client.4397.objecter ms_handle_connect 0x7f6a2806e110 2024-03-25T19:39:38.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.221+0000 7f6a39ffb640 1 -- 172.21.15.44:0/3578461762 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f6a3c0548f0 con 0x7f6a441430c0 2024-03-25T19:39:38.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.221+0000 7f6a39ffb640 10 client.4397.objecter ms_dispatch 0x7f6a44000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:38.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.221+0000 7f6a39ffb640 3 client.4397.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:38.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.221+0000 7f6a39ffb640 3 client.4397.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:38.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.221+0000 7f6a39ffb640 20 client.4397.objecter dump_active .. 0 homeless 2024-03-25T19:39:38.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.221+0000 7f6a39ffb640 1 -- 172.21.15.44:0/3578461762 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6a3c050230 con 0x7f6a441430c0 2024-03-25T19:39:38.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.221+0000 7f6a39ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:38.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.221+0000 7f6a39ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:38.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.477+0000 7f6a4b674640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:38.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.477+0000 7f6a4b674640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:38.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.477+0000 7f6a4b674640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:39:38.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.477+0000 7f6a4b674640 1 -- 172.21.15.44:0/3578461762 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a44074cc0 con 0x7f6a441430c0 2024-03-25T19:39:38.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.636+0000 7f6a39ffb640 1 -- 172.21.15.44:0/3578461762 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f6a3c052330 con 0x7f6a441430c0 2024-03-25T19:39:38.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.636+0000 7f6a39ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:38.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.636+0000 7f6a39ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:38.636 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:39:38.636 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:38.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.638+0000 7f6a4b674640 10 librados: watch_flush enter 2024-03-25T19:39:38.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.638+0000 7f6a4b674640 10 librados: watch_flush exit 2024-03-25T19:39:38.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.638+0000 7f6a4b674640 20 client.4397.objecter shutdown clearing up homeless session... 2024-03-25T19:39:38.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.638+0000 7f6a4b674640 10 client.4397.objecter successfully canceled tick 2024-03-25T19:39:38.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.639+0000 7f6a4b674640 1 -- 172.21.15.44:0/3578461762 >> v1:172.21.15.111:6800/3883215793 conn(0x7f6a2806e110 legacy=0x7f6a280705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:38.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.639+0000 7f6a4b674640 10 monclient: shutdown 2024-03-25T19:39:38.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.639+0000 7f6a4b674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:38.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.639+0000 7f6a4b674640 1 -- 172.21.15.44:0/3578461762 >> v1:172.21.15.44:6790/0 conn(0x7f6a441430c0 legacy=0x7f6a441e64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:38.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.639+0000 7f6a4b674640 1 -- 172.21.15.44:0/3578461762 shutdown_connections 2024-03-25T19:39:38.639 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.639+0000 7f6a4b674640 1 -- 172.21.15.44:0/3578461762 >> 172.21.15.44:0/3578461762 conn(0x7f6a440b7ea0 msgr2=0x7f6a44074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:38.639 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.639+0000 7f6a4b674640 1 -- 172.21.15.44:0/3578461762 shutdown_connections 2024-03-25T19:39:38.639 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.639+0000 7f6a4b674640 1 -- 172.21.15.44:0/3578461762 wait complete. 2024-03-25T19:39:38.639 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:38.639+0000 7f6a4b674640 1 librados: shutdown 2024-03-25T19:39:38.650 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:45.456 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:45.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.565+0000 7f30ef7d6640 10 monclient: get_monmap_and_config 2024-03-25T19:39:45.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.565+0000 7f30ef7d6640 10 monclient: build_initial_monmap 2024-03-25T19:39:45.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.565+0000 7f30ef7d6640 10 monclient: monmap: 2024-03-25T19:39:45.565 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:39:45.565 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:45.565 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:39:45.566432+0000 2024-03-25T19:39:45.566 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:39:45.566432+0000 2024-03-25T19:39:45.566 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:45.566 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:45.566 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:39:45.566 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:39:45.566 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:39:45.566 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:45.567 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.566+0000 7f30ef7d6640 1 Processor -- start 2024-03-25T19:39:45.567 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.566+0000 7f30ef7d6640 1 -- start start 2024-03-25T19:39:45.567 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.566+0000 7f30ef7d6640 10 monclient: init 2024-03-25T19:39:45.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.567+0000 7f30ef7d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:45.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.567+0000 7f30ef7d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:39:45.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.567+0000 7f30ef7d6640 10 monclient(hunting): picked mon.noname-b con 0x7f30e80b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T19:39:45.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.567+0000 7f30ef7d6640 10 monclient(hunting): picked mon.noname-a con 0x7f30e80b1730 addr v1:172.21.15.44:6789/0 2024-03-25T19:39:45.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.567+0000 7f30ef7d6640 10 monclient(hunting): picked mon.noname-c con 0x7f30e814be90 addr v1:172.21.15.111:6789/0 2024-03-25T19:39:45.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.567+0000 7f30ef7d6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e80af940 con 0x7f30e80b1730 2024-03-25T19:39:45.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.567+0000 7f30ef7d6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e80b0a70 con 0x7f30e814be90 2024-03-25T19:39:45.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.567+0000 7f30ef7d6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e80b0d50 con 0x7f30e80b4d60 2024-03-25T19:39:45.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.567+0000 7f30ef7d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:45.568 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.567+0000 7f30ef7d6640 10 monclient(hunting): authenticate will time out at 5649.738182s 2024-03-25T19:39:45.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.574+0000 7f30ed572640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f30e80b4d60 0x7f30e813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:46188/0 (socket says 172.21.15.44:46188) 2024-03-25T19:39:45.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.574+0000 7f30ed572640 1 -- 172.21.15.44:0/1021721902 learned_addr learned my addr 172.21.15.44:0/1021721902 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:39:45.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.583+0000 7f30dffff640 1 -- 172.21.15.44:0/1021721902 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2153472231 0 0) 0x7f30e80b0d50 con 0x7f30e80b4d60 2024-03-25T19:39:45.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.583+0000 7f30dffff640 10 monclient(hunting): _negotiate 2024-03-25T19:39:45.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.583+0000 7f30dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:45.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.583+0000 7f30dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:45.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.583+0000 7f30dffff640 10 monclient(hunting): my global_id is 4400 2024-03-25T19:39:45.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.584+0000 7f30dffff640 1 -- 172.21.15.44:0/1021721902 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30d4003520 con 0x7f30e80b4d60 2024-03-25T19:39:45.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.586+0000 7f30dffff640 1 -- 172.21.15.44:0/1021721902 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487141893 0 0) 0x7f30e80af940 con 0x7f30e80b1730 2024-03-25T19:39:45.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.586+0000 7f30dffff640 10 monclient(hunting): _negotiate 2024-03-25T19:39:45.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.586+0000 7f30dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:45.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.586+0000 7f30dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:45.587 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.586+0000 7f30dffff640 10 monclient(hunting): my global_id is 4431 2024-03-25T19:39:45.587 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.586+0000 7f30dffff640 1 -- 172.21.15.44:0/1021721902 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30e80b0d50 con 0x7f30e80b1730 2024-03-25T19:39:45.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.590+0000 7f30dffff640 1 -- 172.21.15.44:0/1021721902 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542253176 0 0) 0x7f30e80b0a70 con 0x7f30e814be90 2024-03-25T19:39:45.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.590+0000 7f30dffff640 10 monclient(hunting): _negotiate 2024-03-25T19:39:45.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.590+0000 7f30dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:45.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.590+0000 7f30dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:45.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.590+0000 7f30dffff640 10 monclient(hunting): my global_id is 4411 2024-03-25T19:39:45.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.590+0000 7f30dffff640 1 -- 172.21.15.44:0/1021721902 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30e80af940 con 0x7f30e814be90 2024-03-25T19:39:45.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.595+0000 7f30dffff640 1 -- 172.21.15.44:0/1021721902 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4294742144 0 0) 0x7f30d4003520 con 0x7f30e80b4d60 2024-03-25T19:39:45.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.595+0000 7f30dffff640 1 -- 172.21.15.44:0/1021721902 >> v1:172.21.15.111:6789/0 conn(0x7f30e814be90 legacy=0x7f30e814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:45.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.595+0000 7f30dffff640 1 -- 172.21.15.44:0/1021721902 >> v1:172.21.15.44:6789/0 conn(0x7f30e80b1730 legacy=0x7f30e80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:45.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.595+0000 7f30dffff640 10 monclient: _finish_hunting 0 2024-03-25T19:39:45.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.595+0000 7f30dffff640 1 monclient: found mon.noname-b 2024-03-25T19:39:45.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.595+0000 7f30dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:45.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.595+0000 7f30dffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:39:45.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.595+0000 7f30dffff640 1 -- 172.21.15.44:0/1021721902 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e80b1030 con 0x7f30e80b4d60 2024-03-25T19:39:45.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.595+0000 7f30dffff640 10 monclient: _finish_auth 0 2024-03-25T19:39:45.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.595+0000 7f30dffff640 10 monclient: _check_auth_tickets 2024-03-25T19:39:45.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.595+0000 7f30dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:45.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.595+0000 7f30dffff640 1 -- 172.21.15.44:0/1021721902 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f30e4001690 con 0x7f30e80b4d60 2024-03-25T19:39:45.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.595+0000 7f30dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:45.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.595+0000 7f30dffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:45.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.595+0000 7f30dffff640 10 monclient: dump: 2024-03-25T19:39:45.597 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:45.597 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:45.597 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:45.597 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:45.597 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:45.597 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:45.597 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:45.597 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:45.597 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:45.597 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:45.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.595+0000 7f30ef7d6640 5 monclient: authenticate success, global_id 4400 2024-03-25T19:39:45.598 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.595+0000 7f30ef7d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:45.610 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.610+0000 7f30dffff640 1 -- 172.21.15.44:0/1021721902 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30e40017f0 con 0x7f30e80b4d60 2024-03-25T19:39:45.610 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.610+0000 7f30dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:45.610 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.610+0000 7f30dffff640 1 -- 172.21.15.44:0/1021721902 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f30e4001c00 con 0x7f30e80b4d60 2024-03-25T19:39:45.610 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.610+0000 7f30dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:45.610 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.610+0000 7f30dffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:45.610 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.610+0000 7f30dffff640 10 monclient: dump: 2024-03-25T19:39:45.611 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:45.611 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:45.611 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:45.611 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:45.611 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:45.611 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:45.611 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:45.611 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:45.611 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:45.611 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:45.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.610+0000 7f30ef7d6640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:45.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.610+0000 7f30ef7d6640 4 set_mon_vals no callback set 2024-03-25T19:39:45.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.611+0000 7f30ef7d6640 10 monclient: shutdown 2024-03-25T19:39:45.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.611+0000 7f30ef7d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:45.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.611+0000 7f30ef7d6640 1 -- 172.21.15.44:0/1021721902 >> v1:172.21.15.44:6790/0 conn(0x7f30e80b4d60 legacy=0x7f30e813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:45.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.611+0000 7f30ef7d6640 1 -- 172.21.15.44:0/1021721902 shutdown_connections 2024-03-25T19:39:45.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.611+0000 7f30ef7d6640 1 -- 172.21.15.44:0/1021721902 >> 172.21.15.44:0/1021721902 conn(0x7f30e80617a0 msgr2=0x7f30e8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:45.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.611+0000 7f30ef7d6640 1 -- 172.21.15.44:0/1021721902 shutdown_connections 2024-03-25T19:39:45.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.611+0000 7f30ef7d6640 1 -- 172.21.15.44:0/1021721902 wait complete. 2024-03-25T19:39:45.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.612+0000 7f30ef7d6640 10 monclient: build_initial_monmap 2024-03-25T19:39:45.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.612+0000 7f30ef7d6640 10 monclient: monmap: 2024-03-25T19:39:45.614 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:39:45.614 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:45.614 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:39:45.614 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:39:45.614 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:45.614 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:39:45.615 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:39:45.615 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:39:45.615 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:39:45.615 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:45.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.612+0000 7f30ef7d6640 1 librados: starting msgr at 2024-03-25T19:39:45.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.612+0000 7f30ef7d6640 1 librados: starting objecter 2024-03-25T19:39:45.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.612+0000 7f30ef7d6640 1 Processor -- start 2024-03-25T19:39:45.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.612+0000 7f30ef7d6640 1 -- start start 2024-03-25T19:39:45.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.612+0000 7f30ef7d6640 1 librados: setting wanted keys 2024-03-25T19:39:45.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.612+0000 7f30ef7d6640 1 librados: calling monclient init 2024-03-25T19:39:45.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.612+0000 7f30ef7d6640 10 monclient: init 2024-03-25T19:39:45.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.613+0000 7f30ef7d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:45.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.613+0000 7f30ef7d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:39:45.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.613+0000 7f30ef7d6640 10 monclient(hunting): picked mon.noname-b con 0x7f30e80b1730 addr v1:172.21.15.111:6789/0 2024-03-25T19:39:45.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.613+0000 7f30ef7d6640 10 monclient(hunting): picked mon.noname-a con 0x7f30e80b4d60 addr v1:172.21.15.44:6789/0 2024-03-25T19:39:45.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.613+0000 7f30ef7d6640 10 monclient(hunting): picked mon.noname-c con 0x7f30e814be90 addr v1:172.21.15.44:6790/0 2024-03-25T19:39:45.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.613+0000 7f30ef7d6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e8141bf0 con 0x7f30e80b4d60 2024-03-25T19:39:45.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.613+0000 7f30ef7d6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e8141ed0 con 0x7f30e80b1730 2024-03-25T19:39:45.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.613+0000 7f30ef7d6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e81421b0 con 0x7f30e814be90 2024-03-25T19:39:45.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.613+0000 7f30ef7d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:45.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.613+0000 7f30ef7d6640 10 monclient(hunting): authenticate will time out at 5649.784705s 2024-03-25T19:39:45.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.620+0000 7f30edd73640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f30e814be90 0x7f30e81341f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:46210/0 (socket says 172.21.15.44:46210) 2024-03-25T19:39:45.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.620+0000 7f30edd73640 1 -- 172.21.15.44:0/903661616 learned_addr learned my addr 172.21.15.44:0/903661616 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:39:45.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.620+0000 7f30ddffb640 10 client.?.objecter ms_handle_connect 0x7f30e814be90 2024-03-25T19:39:45.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.620+0000 7f30ddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:45.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.623+0000 7f30ddffb640 10 client.?.objecter ms_handle_connect 0x7f30e80b4d60 2024-03-25T19:39:45.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.623+0000 7f30ddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:45.628 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.628+0000 7f30ddffb640 1 -- 172.21.15.44:0/903661616 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659722942 0 0) 0x7f30e81421b0 con 0x7f30e814be90 2024-03-25T19:39:45.628 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.628+0000 7f30ddffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:45.628 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.628+0000 7f30ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:45.628 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.628+0000 7f30ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:45.628 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.628+0000 7f30ddffb640 10 monclient(hunting): my global_id is 4403 2024-03-25T19:39:45.628 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.628+0000 7f30ddffb640 1 -- 172.21.15.44:0/903661616 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30cc004eb0 con 0x7f30e814be90 2024-03-25T19:39:45.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.637+0000 7f30ddffb640 10 client.?.objecter ms_handle_connect 0x7f30e80b1730 2024-03-25T19:39:45.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.637+0000 7f30ddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:45.639 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.639+0000 7f30ddffb640 1 -- 172.21.15.44:0/903661616 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3404936208 0 0) 0x7f30e8141bf0 con 0x7f30e80b4d60 2024-03-25T19:39:45.639 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.639+0000 7f30ddffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:45.639 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.639+0000 7f30ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:45.639 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.639+0000 7f30ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:45.639 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.639+0000 7f30ddffb640 10 monclient(hunting): my global_id is 4434 2024-03-25T19:39:45.639 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.639+0000 7f30ddffb640 1 -- 172.21.15.44:0/903661616 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30e81421b0 con 0x7f30e80b4d60 2024-03-25T19:39:45.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.643+0000 7f30ddffb640 1 -- 172.21.15.44:0/903661616 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1742625576 0 0) 0x7f30cc004eb0 con 0x7f30e814be90 2024-03-25T19:39:45.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.643+0000 7f30ddffb640 1 -- 172.21.15.44:0/903661616 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30e8141bf0 con 0x7f30e814be90 2024-03-25T19:39:45.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.643+0000 7f30ddffb640 1 -- 172.21.15.44:0/903661616 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f30e0001c70 con 0x7f30e814be90 2024-03-25T19:39:45.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.643+0000 7f30ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:45.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.643+0000 7f30ddffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:45.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.643+0000 7f30ddffb640 10 monclient(hunting): dump: 2024-03-25T19:39:45.644 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:45.644 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:45.644 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:45.644 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:45.644 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:45.644 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:45.644 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:45.644 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:45.644 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:45.644 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:45.644 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.643+0000 7f30ddffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:45.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.648+0000 7f30ddffb640 1 -- 172.21.15.44:0/903661616 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 874013310 0 0) 0x7f30e81421b0 con 0x7f30e80b4d60 2024-03-25T19:39:45.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.649+0000 7f30ddffb640 1 -- 172.21.15.44:0/903661616 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30cc004eb0 con 0x7f30e80b4d60 2024-03-25T19:39:45.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.649+0000 7f30ddffb640 1 -- 172.21.15.44:0/903661616 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f30d8001970 con 0x7f30e80b4d60 2024-03-25T19:39:45.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.649+0000 7f30ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:45.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.649+0000 7f30ddffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:45.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.649+0000 7f30ddffb640 10 monclient(hunting): dump: 2024-03-25T19:39:45.649 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:45.649 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:45.649 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:45.649 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:45.649 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:45.649 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:45.649 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:45.649 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:45.649 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:45.649 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:45.650 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.650+0000 7f30ddffb640 1 -- 172.21.15.44:0/903661616 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063020178 0 0) 0x7f30e8141ed0 con 0x7f30e80b1730 2024-03-25T19:39:45.650 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.650+0000 7f30ddffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:45.650 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.650+0000 7f30ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:45.650 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.650+0000 7f30ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:45.650 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.650+0000 7f30ddffb640 10 monclient(hunting): my global_id is 4414 2024-03-25T19:39:45.650 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.650+0000 7f30ddffb640 1 -- 172.21.15.44:0/903661616 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30e81421b0 con 0x7f30e80b1730 2024-03-25T19:39:45.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.655+0000 7f30ddffb640 1 -- 172.21.15.44:0/903661616 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4088493679 0 0) 0x7f30e8141bf0 con 0x7f30e814be90 2024-03-25T19:39:45.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.655+0000 7f30ddffb640 1 -- 172.21.15.44:0/903661616 >> v1:172.21.15.111:6789/0 conn(0x7f30e80b1730 legacy=0x7f30e81425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:45.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.655+0000 7f30ddffb640 1 -- 172.21.15.44:0/903661616 >> v1:172.21.15.44:6789/0 conn(0x7f30e80b4d60 legacy=0x7f30e8133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:45.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.655+0000 7f30ddffb640 10 monclient: _finish_hunting 0 2024-03-25T19:39:45.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.655+0000 7f30ddffb640 1 monclient: found mon.c 2024-03-25T19:39:45.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.655+0000 7f30ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:45.657 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.656+0000 7f30ddffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:39:45.657 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.656+0000 7f30ddffb640 1 -- 172.21.15.44:0/903661616 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e8140f00 con 0x7f30e814be90 2024-03-25T19:39:45.657 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.656+0000 7f30ddffb640 10 monclient: _finish_auth 0 2024-03-25T19:39:45.657 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.656+0000 7f30ddffb640 10 monclient: _check_auth_tickets 2024-03-25T19:39:45.657 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.656+0000 7f30ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:45.657 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.656+0000 7f30ef7d6640 5 monclient: authenticate success, global_id 4403 2024-03-25T19:39:45.657 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.656+0000 7f30ef7d6640 10 monclient: _renew_subs 2024-03-25T19:39:45.657 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.656+0000 7f30ef7d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:39:45.657 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.656+0000 7f30ef7d6640 1 -- 172.21.15.44:0/903661616 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e81406f0 con 0x7f30e814be90 2024-03-25T19:39:45.657 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.656+0000 7f30ef7d6640 10 client.4403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:45.657 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.656+0000 7f30ef7d6640 10 monclient: _renew_subs 2024-03-25T19:39:45.657 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.656+0000 7f30ef7d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:39:45.657 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.656+0000 7f30ef7d6640 1 -- 172.21.15.44:0/903661616 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e8140380 con 0x7f30e814be90 2024-03-25T19:39:45.657 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.656+0000 7f30ef7d6640 1 librados: init done 2024-03-25T19:39:45.658 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.657+0000 7f30ef7d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:45.658 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.657+0000 7f30ef7d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:45.658 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.657+0000 7f30ef7d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:39:45.658 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.657+0000 7f30ef7d6640 1 -- 172.21.15.44:0/903661616 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e81401f0 con 0x7f30e814be90 2024-03-25T19:39:45.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.982+0000 7f30ddffb640 1 -- 172.21.15.44:0/903661616 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30e0003990 con 0x7f30e814be90 2024-03-25T19:39:45.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.982+0000 7f30ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:45.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.982+0000 7f30ddffb640 1 -- 172.21.15.44:0/903661616 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f30e0003da0 con 0x7f30e814be90 2024-03-25T19:39:45.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.982+0000 7f30ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:45.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.982+0000 7f30ddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:45.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.982+0000 7f30ddffb640 10 monclient: dump: 2024-03-25T19:39:45.982 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:45.982 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:45.983 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:45.983 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:45.983 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:45.983 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:45.983 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:45.983 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:45.983 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:45.983 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:45.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.982+0000 7f30df7fe640 4 set_mon_vals no callback set 2024-03-25T19:39:45.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.983+0000 7f30ddffb640 1 -- 172.21.15.44:0/903661616 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f30e0003fc0 con 0x7f30e814be90 2024-03-25T19:39:45.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.984+0000 7f30ed572640 10 monclient: get_auth_request con 0x7f30cc06df50 auth_method 0 2024-03-25T19:39:45.984 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.984+0000 7f30ddffb640 10 client.4403.objecter ms_handle_connect 0x7f30cc06df50 2024-03-25T19:39:45.992 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.992+0000 7f30ddffb640 1 -- 172.21.15.44:0/903661616 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f30e00549d0 con 0x7f30e814be90 2024-03-25T19:39:45.992 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.992+0000 7f30ddffb640 10 client.4403.objecter ms_dispatch 0x7f30e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:45.992 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.992+0000 7f30ddffb640 3 client.4403.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:45.992 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.992+0000 7f30ddffb640 3 client.4403.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:45.992 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.992+0000 7f30ddffb640 20 client.4403.objecter dump_active .. 0 homeless 2024-03-25T19:39:45.992 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.992+0000 7f30ddffb640 1 -- 172.21.15.44:0/903661616 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f30e00504e0 con 0x7f30e814be90 2024-03-25T19:39:45.992 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.992+0000 7f30ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:45.992 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:45.992+0000 7f30ddffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:46.247 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:46.247+0000 7f30ef7d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:46.247 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:46.247+0000 7f30ef7d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:46.247 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:46.247+0000 7f30ef7d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:39:46.247 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:46.247+0000 7f30ef7d6640 1 -- 172.21.15.44:0/903661616 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30e80627b0 con 0x7f30e814be90 2024-03-25T19:39:46.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:46.280+0000 7f30ddffb640 1 -- 172.21.15.44:0/903661616 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f30e0052410 con 0x7f30e814be90 2024-03-25T19:39:46.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:46.280+0000 7f30ddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:46.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:46.280+0000 7f30ddffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:46.280 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:39:46.280 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:46.283 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:46.282+0000 7f30ef7d6640 10 librados: watch_flush enter 2024-03-25T19:39:46.283 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:46.282+0000 7f30ef7d6640 10 librados: watch_flush exit 2024-03-25T19:39:46.283 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:46.282+0000 7f30ef7d6640 20 client.4403.objecter shutdown clearing up homeless session... 2024-03-25T19:39:46.283 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:46.282+0000 7f30ef7d6640 10 client.4403.objecter successfully canceled tick 2024-03-25T19:39:46.283 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:46.282+0000 7f30ef7d6640 1 -- 172.21.15.44:0/903661616 >> v1:172.21.15.111:6800/3883215793 conn(0x7f30cc06df50 legacy=0x7f30cc070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:46.283 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:46.282+0000 7f30ef7d6640 10 monclient: shutdown 2024-03-25T19:39:46.283 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:46.282+0000 7f30ef7d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:46.283 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:46.282+0000 7f30ef7d6640 1 -- 172.21.15.44:0/903661616 >> v1:172.21.15.44:6790/0 conn(0x7f30e814be90 legacy=0x7f30e81341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:46.283 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:46.283+0000 7f30ef7d6640 1 -- 172.21.15.44:0/903661616 shutdown_connections 2024-03-25T19:39:46.283 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:46.283+0000 7f30ef7d6640 1 -- 172.21.15.44:0/903661616 >> 172.21.15.44:0/903661616 conn(0x7f30e80617a0 msgr2=0x7f30e813c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:46.283 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:46.283+0000 7f30ef7d6640 1 -- 172.21.15.44:0/903661616 shutdown_connections 2024-03-25T19:39:46.284 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:46.283+0000 7f30ef7d6640 1 -- 172.21.15.44:0/903661616 wait complete. 2024-03-25T19:39:46.284 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:46.283+0000 7f30ef7d6640 1 librados: shutdown 2024-03-25T19:39:46.294 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:53.100 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:53.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.219+0000 7f5430dc8640 10 monclient: get_monmap_and_config 2024-03-25T19:39:53.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.219+0000 7f5430dc8640 10 monclient: build_initial_monmap 2024-03-25T19:39:53.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.219+0000 7f5430dc8640 10 monclient: monmap: 2024-03-25T19:39:53.219 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:39:53.219 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:53.219 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:39:53.220487+0000 2024-03-25T19:39:53.219 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:39:53.220487+0000 2024-03-25T19:39:53.219 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:53.219 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:53.220 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:39:53.220 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:39:53.220 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:39:53.220 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:53.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.220+0000 7f5430dc8640 1 Processor -- start 2024-03-25T19:39:53.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.220+0000 7f5430dc8640 1 -- start start 2024-03-25T19:39:53.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.220+0000 7f5430dc8640 10 monclient: init 2024-03-25T19:39:53.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.221+0000 7f5430dc8640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:53.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.221+0000 7f5430dc8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:39:53.222 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.221+0000 7f5430dc8640 10 monclient(hunting): picked mon.noname-a con 0x7f542c0b4d60 addr v1:172.21.15.44:6789/0 2024-03-25T19:39:53.222 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.221+0000 7f5430dc8640 10 monclient(hunting): picked mon.noname-c con 0x7f542c0b1730 addr v1:172.21.15.111:6789/0 2024-03-25T19:39:53.222 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.221+0000 7f5430dc8640 10 monclient(hunting): picked mon.noname-b con 0x7f542c14be90 addr v1:172.21.15.44:6790/0 2024-03-25T19:39:53.222 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.221+0000 7f5430dc8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542c0af940 con 0x7f542c0b4d60 2024-03-25T19:39:53.222 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.221+0000 7f5430dc8640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542c0b0a70 con 0x7f542c0b1730 2024-03-25T19:39:53.222 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.221+0000 7f5430dc8640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542c0b0d50 con 0x7f542c14be90 2024-03-25T19:39:53.222 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.221+0000 7f5430dc8640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:53.222 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.221+0000 7f5430dc8640 10 monclient(hunting): authenticate will time out at 5657.392360s 2024-03-25T19:39:53.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.232+0000 7f5423fff640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f542c0b1730 0x7f542c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:34064/0 (socket says 172.21.15.44:34064) 2024-03-25T19:39:53.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.232+0000 7f5423fff640 1 -- 172.21.15.44:0/1760957024 learned_addr learned my addr 172.21.15.44:0/1760957024 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:39:53.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.239+0000 7f5429d9b640 1 -- 172.21.15.44:0/1760957024 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2833013540 0 0) 0x7f542c0b0a70 con 0x7f542c0b1730 2024-03-25T19:39:53.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.239+0000 7f5429d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:39:53.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.239+0000 7f5429d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:53.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.239+0000 7f5429d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:53.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.239+0000 7f5429d9b640 10 monclient(hunting): my global_id is 4417 2024-03-25T19:39:53.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.239+0000 7f5429d9b640 1 -- 172.21.15.44:0/1760957024 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5418003520 con 0x7f542c0b1730 2024-03-25T19:39:53.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.248+0000 7f5429d9b640 1 -- 172.21.15.44:0/1760957024 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356955734 0 0) 0x7f542c0b0d50 con 0x7f542c14be90 2024-03-25T19:39:53.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.248+0000 7f5429d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:39:53.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.248+0000 7f5429d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:53.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.248+0000 7f5429d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:53.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.248+0000 7f5429d9b640 10 monclient(hunting): my global_id is 4406 2024-03-25T19:39:53.248 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.248+0000 7f5429d9b640 1 -- 172.21.15.44:0/1760957024 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f542c0b0a70 con 0x7f542c14be90 2024-03-25T19:39:53.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.250+0000 7f5429d9b640 1 -- 172.21.15.44:0/1760957024 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263408166 0 0) 0x7f542c0af940 con 0x7f542c0b4d60 2024-03-25T19:39:53.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.250+0000 7f5429d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:39:53.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.250+0000 7f5429d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:53.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.250+0000 7f5429d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:53.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.250+0000 7f5429d9b640 10 monclient(hunting): my global_id is 4437 2024-03-25T19:39:53.250 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.250+0000 7f5429d9b640 1 -- 172.21.15.44:0/1760957024 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f542c0b0d50 con 0x7f542c0b4d60 2024-03-25T19:39:53.255 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.255+0000 7f5429d9b640 1 -- 172.21.15.44:0/1760957024 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3551450772 0 0) 0x7f5418003520 con 0x7f542c0b1730 2024-03-25T19:39:53.255 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.255+0000 7f5429d9b640 1 -- 172.21.15.44:0/1760957024 >> v1:172.21.15.44:6790/0 conn(0x7f542c14be90 legacy=0x7f542c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:53.255 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.255+0000 7f5429d9b640 1 -- 172.21.15.44:0/1760957024 >> v1:172.21.15.44:6789/0 conn(0x7f542c0b4d60 legacy=0x7f542c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:53.255 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.255+0000 7f5429d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:39:53.255 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.255+0000 7f5429d9b640 1 monclient: found mon.noname-c 2024-03-25T19:39:53.255 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.255+0000 7f5429d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:53.255 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.255+0000 7f5429d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:39:53.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.255+0000 7f5429d9b640 1 -- 172.21.15.44:0/1760957024 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f542c0b1030 con 0x7f542c0b1730 2024-03-25T19:39:53.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.255+0000 7f5429d9b640 10 monclient: _finish_auth 0 2024-03-25T19:39:53.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.255+0000 7f5429d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:39:53.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.255+0000 7f5429d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:53.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.255+0000 7f5429d9b640 1 -- 172.21.15.44:0/1760957024 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f5414001690 con 0x7f542c0b1730 2024-03-25T19:39:53.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.255+0000 7f5429d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:53.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.255+0000 7f5429d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:53.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.255+0000 7f5429d9b640 10 monclient: dump: 2024-03-25T19:39:53.256 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:53.256 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:53.256 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:53.256 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:53.256 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:53.256 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:53.256 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:53.256 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:53.257 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:53.257 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:53.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.255+0000 7f5430dc8640 5 monclient: authenticate success, global_id 4417 2024-03-25T19:39:53.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.255+0000 7f5430dc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:53.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.280+0000 7f5429d9b640 1 -- 172.21.15.44:0/1760957024 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54140018b0 con 0x7f542c0b1730 2024-03-25T19:39:53.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.280+0000 7f5429d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:53.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.280+0000 7f5429d9b640 1 -- 172.21.15.44:0/1760957024 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f5414001c40 con 0x7f542c0b1730 2024-03-25T19:39:53.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.280+0000 7f5429d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:53.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.280+0000 7f5429d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:53.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.280+0000 7f5429d9b640 10 monclient: dump: 2024-03-25T19:39:53.281 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:53.281 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:53.281 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:53.281 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:53.281 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:53.281 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:53.281 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:53.281 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:53.281 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:53.282 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:53.282 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.281+0000 7f5430dc8640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:53.282 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.281+0000 7f5430dc8640 4 set_mon_vals no callback set 2024-03-25T19:39:53.283 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.281+0000 7f5430dc8640 10 monclient: shutdown 2024-03-25T19:39:53.284 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.281+0000 7f5430dc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:53.284 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.281+0000 7f5430dc8640 1 -- 172.21.15.44:0/1760957024 >> v1:172.21.15.111:6789/0 conn(0x7f542c0b1730 legacy=0x7f542c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:53.284 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.281+0000 7f5430dc8640 1 -- 172.21.15.44:0/1760957024 shutdown_connections 2024-03-25T19:39:53.284 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.281+0000 7f5430dc8640 1 -- 172.21.15.44:0/1760957024 >> 172.21.15.44:0/1760957024 conn(0x7f542c0617a0 msgr2=0x7f542c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:53.284 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.281+0000 7f5430dc8640 1 -- 172.21.15.44:0/1760957024 shutdown_connections 2024-03-25T19:39:53.284 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.281+0000 7f5430dc8640 1 -- 172.21.15.44:0/1760957024 wait complete. 2024-03-25T19:39:53.284 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.282+0000 7f5430dc8640 10 monclient: build_initial_monmap 2024-03-25T19:39:53.284 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.282+0000 7f5430dc8640 10 monclient: monmap: 2024-03-25T19:39:53.284 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:39:53.284 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:53.285 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:39:53.285 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:39:53.285 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:53.285 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:39:53.285 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:39:53.285 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:39:53.285 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:39:53.285 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:53.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.283+0000 7f5430dc8640 1 librados: starting msgr at 2024-03-25T19:39:53.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.283+0000 7f5430dc8640 1 librados: starting objecter 2024-03-25T19:39:53.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.283+0000 7f5430dc8640 1 Processor -- start 2024-03-25T19:39:53.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.283+0000 7f5430dc8640 1 -- start start 2024-03-25T19:39:53.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.283+0000 7f5430dc8640 1 librados: setting wanted keys 2024-03-25T19:39:53.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.283+0000 7f5430dc8640 1 librados: calling monclient init 2024-03-25T19:39:53.286 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.283+0000 7f5430dc8640 10 monclient: init 2024-03-25T19:39:53.286 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.283+0000 7f5430dc8640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:53.286 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.283+0000 7f5430dc8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:39:53.286 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.283+0000 7f5430dc8640 10 monclient(hunting): picked mon.noname-a con 0x7f542c0b1730 addr v1:172.21.15.44:6789/0 2024-03-25T19:39:53.286 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.283+0000 7f5430dc8640 10 monclient(hunting): picked mon.noname-c con 0x7f542c0b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T19:39:53.286 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.283+0000 7f5430dc8640 10 monclient(hunting): picked mon.noname-b con 0x7f542c14be90 addr v1:172.21.15.111:6789/0 2024-03-25T19:39:53.286 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.283+0000 7f5430dc8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542c134990 con 0x7f542c0b1730 2024-03-25T19:39:53.286 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.283+0000 7f5430dc8640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542c141b50 con 0x7f542c14be90 2024-03-25T19:39:53.286 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.283+0000 7f5430dc8640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542c141e30 con 0x7f542c0b4d60 2024-03-25T19:39:53.286 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.283+0000 7f5430dc8640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:53.286 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.283+0000 7f5430dc8640 10 monclient(hunting): authenticate will time out at 5657.454741s 2024-03-25T19:39:53.288 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.289+0000 7f542ad9d640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f542c14be90 0x7f542c134280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:34120/0 (socket says 172.21.15.44:34120) 2024-03-25T19:39:53.288 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.289+0000 7f542ad9d640 1 -- 172.21.15.44:0/433554632 learned_addr learned my addr 172.21.15.44:0/433554632 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:39:53.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.291+0000 7f5422ffd640 10 client.?.objecter ms_handle_connect 0x7f542c14be90 2024-03-25T19:39:53.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.291+0000 7f5422ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:53.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.298+0000 7f5422ffd640 1 -- 172.21.15.44:0/433554632 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552310271 0 0) 0x7f542c141b50 con 0x7f542c14be90 2024-03-25T19:39:53.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.298+0000 7f5422ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:53.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.298+0000 7f5422ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:53.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.298+0000 7f5422ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:53.299 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.298+0000 7f5422ffd640 10 monclient(hunting): my global_id is 4420 2024-03-25T19:39:53.299 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.298+0000 7f5422ffd640 1 -- 172.21.15.44:0/433554632 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f540c004eb0 con 0x7f542c14be90 2024-03-25T19:39:53.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.308+0000 7f5422ffd640 10 client.?.objecter ms_handle_connect 0x7f542c0b1730 2024-03-25T19:39:53.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.308+0000 7f5422ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:53.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.309+0000 7f5422ffd640 1 -- 172.21.15.44:0/433554632 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 110197198 0 0) 0x7f540c004eb0 con 0x7f542c14be90 2024-03-25T19:39:53.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.309+0000 7f5422ffd640 1 -- 172.21.15.44:0/433554632 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f542c141b50 con 0x7f542c14be90 2024-03-25T19:39:53.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.309+0000 7f5422ffd640 1 -- 172.21.15.44:0/433554632 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f5424001c90 con 0x7f542c14be90 2024-03-25T19:39:53.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.309+0000 7f5422ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:53.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.309+0000 7f5422ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:53.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.309+0000 7f5422ffd640 10 monclient(hunting): dump: 2024-03-25T19:39:53.310 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:53.310 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:53.310 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:53.310 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:53.310 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:53.310 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:53.310 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:53.310 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:53.311 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:53.311 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:53.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.309+0000 7f5422ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:53.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.313+0000 7f5422ffd640 10 client.?.objecter ms_handle_connect 0x7f542c0b4d60 2024-03-25T19:39:53.313 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.313+0000 7f5422ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:53.319 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.319+0000 7f5422ffd640 1 -- 172.21.15.44:0/433554632 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1905624774 0 0) 0x7f542c141e30 con 0x7f542c0b4d60 2024-03-25T19:39:53.319 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.319+0000 7f5422ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:53.319 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.319+0000 7f5422ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:53.319 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.319+0000 7f5422ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:53.319 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.319+0000 7f5422ffd640 10 monclient(hunting): my global_id is 4409 2024-03-25T19:39:53.319 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.319+0000 7f5422ffd640 1 -- 172.21.15.44:0/433554632 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f540c004eb0 con 0x7f542c0b4d60 2024-03-25T19:39:53.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.320+0000 7f5422ffd640 1 -- 172.21.15.44:0/433554632 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 170111207 0 0) 0x7f542c141b50 con 0x7f542c14be90 2024-03-25T19:39:53.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.320+0000 7f5422ffd640 1 -- 172.21.15.44:0/433554632 >> v1:172.21.15.44:6790/0 conn(0x7f542c0b4d60 legacy=0x7f542c133b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:53.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.320+0000 7f5422ffd640 1 -- 172.21.15.44:0/433554632 >> v1:172.21.15.44:6789/0 conn(0x7f542c0b1730 legacy=0x7f542c142250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:53.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.320+0000 7f5422ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:39:53.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.320+0000 7f5422ffd640 1 monclient: found mon.b 2024-03-25T19:39:53.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.320+0000 7f5422ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:53.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.320+0000 7f5422ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:39:53.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.320+0000 7f5422ffd640 1 -- 172.21.15.44:0/433554632 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f542c1406d0 con 0x7f542c14be90 2024-03-25T19:39:53.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.320+0000 7f5422ffd640 10 monclient: _finish_auth 0 2024-03-25T19:39:53.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.320+0000 7f5422ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:39:53.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.320+0000 7f5422ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:53.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.320+0000 7f5430dc8640 5 monclient: authenticate success, global_id 4420 2024-03-25T19:39:53.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.320+0000 7f5430dc8640 10 monclient: _renew_subs 2024-03-25T19:39:53.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.320+0000 7f5430dc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:39:53.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.320+0000 7f5430dc8640 1 -- 172.21.15.44:0/433554632 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f542c140380 con 0x7f542c14be90 2024-03-25T19:39:53.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.320+0000 7f5430dc8640 10 client.4420.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:53.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.320+0000 7f5430dc8640 10 monclient: _renew_subs 2024-03-25T19:39:53.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.320+0000 7f5430dc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:39:53.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.320+0000 7f5430dc8640 1 -- 172.21.15.44:0/433554632 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f542c13bb10 con 0x7f542c14be90 2024-03-25T19:39:53.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.320+0000 7f5430dc8640 1 librados: init done 2024-03-25T19:39:53.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.321+0000 7f5430dc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:53.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.321+0000 7f5430dc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:53.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.321+0000 7f5430dc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:39:53.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.321+0000 7f5430dc8640 1 -- 172.21.15.44:0/433554632 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f542c13bdc0 con 0x7f542c14be90 2024-03-25T19:39:53.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.590+0000 7f5422ffd640 1 -- 172.21.15.44:0/433554632 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5424003970 con 0x7f542c14be90 2024-03-25T19:39:53.591 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.590+0000 7f5422ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:53.591 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.590+0000 7f5422ffd640 1 -- 172.21.15.44:0/433554632 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f5424003d00 con 0x7f542c14be90 2024-03-25T19:39:53.591 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.590+0000 7f5422ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:53.591 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.590+0000 7f5422ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:53.591 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.590+0000 7f5422ffd640 10 monclient: dump: 2024-03-25T19:39:53.591 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:39:53.591 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:39:53.591 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:53.591 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:39:53.591 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:53.591 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:39:53.591 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:39:53.591 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:39:53.591 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:39:53.592 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:39:53.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.590+0000 7f542959a640 4 set_mon_vals no callback set 2024-03-25T19:39:53.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.591+0000 7f5422ffd640 1 -- 172.21.15.44:0/433554632 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f5424003f20 con 0x7f542c14be90 2024-03-25T19:39:53.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.592+0000 7f542a59c640 10 monclient: get_auth_request con 0x7f540c06de20 auth_method 0 2024-03-25T19:39:53.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.592+0000 7f5422ffd640 10 client.4420.objecter ms_handle_connect 0x7f540c06de20 2024-03-25T19:39:53.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.596+0000 7f5422ffd640 1 -- 172.21.15.44:0/433554632 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f5424054a20 con 0x7f542c14be90 2024-03-25T19:39:53.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.596+0000 7f5422ffd640 10 client.4420.objecter ms_dispatch 0x7f542c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:53.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.596+0000 7f5422ffd640 3 client.4420.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:53.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.596+0000 7f5422ffd640 3 client.4420.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:53.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.596+0000 7f5422ffd640 20 client.4420.objecter dump_active .. 0 homeless 2024-03-25T19:39:53.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.600+0000 7f5422ffd640 1 -- 172.21.15.44:0/433554632 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5424050530 con 0x7f542c14be90 2024-03-25T19:39:53.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.600+0000 7f5422ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:53.599 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.600+0000 7f5422ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:53.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.818+0000 7f5430dc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:53.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.818+0000 7f5430dc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:53.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.818+0000 7f5430dc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:39:53.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.818+0000 7f5430dc8640 1 -- 172.21.15.44:0/433554632 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f542c140510 con 0x7f542c14be90 2024-03-25T19:39:53.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.836+0000 7f5422ffd640 1 -- 172.21.15.44:0/433554632 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f5424052460 con 0x7f542c14be90 2024-03-25T19:39:53.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.836+0000 7f5422ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:53.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.836+0000 7f5422ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:53.836 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:39:53.836 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:53.838 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.838+0000 7f5430dc8640 10 librados: watch_flush enter 2024-03-25T19:39:53.838 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.838+0000 7f5430dc8640 10 librados: watch_flush exit 2024-03-25T19:39:53.838 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.838+0000 7f5430dc8640 20 client.4420.objecter shutdown clearing up homeless session... 2024-03-25T19:39:53.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.838+0000 7f5430dc8640 10 client.4420.objecter successfully canceled tick 2024-03-25T19:39:53.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.838+0000 7f5430dc8640 1 -- 172.21.15.44:0/433554632 >> v1:172.21.15.111:6800/3883215793 conn(0x7f540c06de20 legacy=0x7f540c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:53.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.838+0000 7f5430dc8640 10 monclient: shutdown 2024-03-25T19:39:53.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.838+0000 7f5430dc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:53.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.838+0000 7f5430dc8640 1 -- 172.21.15.44:0/433554632 >> v1:172.21.15.111:6789/0 conn(0x7f542c14be90 legacy=0x7f542c134280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:53.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.838+0000 7f5430dc8640 1 -- 172.21.15.44:0/433554632 shutdown_connections 2024-03-25T19:39:53.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.838+0000 7f5430dc8640 1 -- 172.21.15.44:0/433554632 >> 172.21.15.44:0/433554632 conn(0x7f542c0617a0 msgr2=0x7f542c13c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:53.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.838+0000 7f5430dc8640 1 -- 172.21.15.44:0/433554632 shutdown_connections 2024-03-25T19:39:53.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.838+0000 7f5430dc8640 1 -- 172.21.15.44:0/433554632 wait complete. 2024-03-25T19:39:53.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:39:53.838+0000 7f5430dc8640 1 librados: shutdown 2024-03-25T19:39:53.850 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:59.414 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:04:50.316 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:59.421 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:04:50.323 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:59.466 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:04:50.367 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:59.479 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:04:50.358 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:59.487 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:04:50.367 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:59.508 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:04:50.388 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:59.515 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:04:50.416 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:59.528 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:04:50.430 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:59.548 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:04:50.450 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:00.658 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:00.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.765+0000 7f087d5c6640 10 monclient: get_monmap_and_config 2024-03-25T19:40:00.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.765+0000 7f087d5c6640 10 monclient: build_initial_monmap 2024-03-25T19:40:00.766 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.765+0000 7f087d5c6640 10 monclient: monmap: 2024-03-25T19:40:00.766 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:40:00.766 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:00.766 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:40:00.766369+0000 2024-03-25T19:40:00.766 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:40:00.766369+0000 2024-03-25T19:40:00.766 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:00.766 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:00.766 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:40:00.766 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:40:00.766 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:40:00.766 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:00.766 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.765+0000 7f087d5c6640 1 Processor -- start 2024-03-25T19:40:00.767 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.765+0000 7f087d5c6640 1 -- start start 2024-03-25T19:40:00.767 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.765+0000 7f087d5c6640 10 monclient: init 2024-03-25T19:40:00.767 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.766+0000 7f087d5c6640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:00.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.766+0000 7f087d5c6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:00.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.766+0000 7f087d5c6640 10 monclient(hunting): picked mon.noname-c con 0x7f08780a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:40:00.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.766+0000 7f087d5c6640 10 monclient(hunting): picked mon.noname-b con 0x7f087813f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:40:00.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.766+0000 7f087d5c6640 10 monclient(hunting): picked mon.noname-a con 0x7f08781430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:40:00.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.766+0000 7f087d5c6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0878142520 con 0x7f08781430c0 2024-03-25T19:40:00.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.766+0000 7f087d5c6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08780b28a0 con 0x7f08780a98f0 2024-03-25T19:40:00.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.766+0000 7f087d5c6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08780b2a30 con 0x7f087813f710 2024-03-25T19:40:00.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.766+0000 7f087d5c6640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:00.768 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.766+0000 7f087d5c6640 10 monclient(hunting): authenticate will time out at 5664.937323s 2024-03-25T19:40:00.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.809+0000 7f0876ffd640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f08780a98f0 0x7f08780a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:56038/0 (socket says 172.21.15.44:56038) 2024-03-25T19:40:00.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.809+0000 7f0876ffd640 1 -- 172.21.15.44:0/3920010656 learned_addr learned my addr 172.21.15.44:0/3920010656 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:40:00.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.821+0000 7f0875ffb640 1 -- 172.21.15.44:0/3920010656 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153540100 0 0) 0x7f08780b2a30 con 0x7f087813f710 2024-03-25T19:40:00.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.821+0000 7f0875ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:00.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.821+0000 7f0875ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:00.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.821+0000 7f0875ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:00.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.821+0000 7f0875ffb640 10 monclient(hunting): my global_id is 4412 2024-03-25T19:40:00.822 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.821+0000 7f0875ffb640 1 -- 172.21.15.44:0/3920010656 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0860003520 con 0x7f087813f710 2024-03-25T19:40:00.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.832+0000 7f0875ffb640 1 -- 172.21.15.44:0/3920010656 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2278081123 0 0) 0x7f08780b28a0 con 0x7f08780a98f0 2024-03-25T19:40:00.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.832+0000 7f0875ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:00.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.832+0000 7f0875ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:00.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.832+0000 7f0875ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:00.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.832+0000 7f0875ffb640 10 monclient(hunting): my global_id is 4423 2024-03-25T19:40:00.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.832+0000 7f0875ffb640 1 -- 172.21.15.44:0/3920010656 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08780b2a30 con 0x7f08780a98f0 2024-03-25T19:40:00.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.839+0000 7f0875ffb640 1 -- 172.21.15.44:0/3920010656 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222483031 0 0) 0x7f0878142520 con 0x7f08781430c0 2024-03-25T19:40:00.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.839+0000 7f0875ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:00.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.839+0000 7f0875ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:00.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.839+0000 7f0875ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:00.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.839+0000 7f0875ffb640 10 monclient(hunting): my global_id is 4443 2024-03-25T19:40:00.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.839+0000 7f0875ffb640 1 -- 172.21.15.44:0/3920010656 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08780b28a0 con 0x7f08781430c0 2024-03-25T19:40:00.841 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.841+0000 7f0875ffb640 1 -- 172.21.15.44:0/3920010656 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3060262199 0 0) 0x7f0860003520 con 0x7f087813f710 2024-03-25T19:40:00.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.841+0000 7f0875ffb640 1 -- 172.21.15.44:0/3920010656 >> v1:172.21.15.111:6789/0 conn(0x7f08780a98f0 legacy=0x7f08780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:00.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.841+0000 7f0875ffb640 1 -- 172.21.15.44:0/3920010656 >> v1:172.21.15.44:6789/0 conn(0x7f08781430c0 legacy=0x7f087814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:00.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.841+0000 7f0875ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:40:00.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.841+0000 7f0875ffb640 1 monclient: found mon.noname-b 2024-03-25T19:40:00.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.841+0000 7f0875ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:00.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.841+0000 7f0875ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:40:00.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.841+0000 7f0875ffb640 1 -- 172.21.15.44:0/3920010656 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08780b44c0 con 0x7f087813f710 2024-03-25T19:40:00.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.841+0000 7f0875ffb640 10 monclient: _finish_auth 0 2024-03-25T19:40:00.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.841+0000 7f0875ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:40:00.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.841+0000 7f0875ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:00.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.841+0000 7f0875ffb640 1 -- 172.21.15.44:0/3920010656 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f086c001690 con 0x7f087813f710 2024-03-25T19:40:00.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.841+0000 7f087d5c6640 5 monclient: authenticate success, global_id 4412 2024-03-25T19:40:00.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.841+0000 7f087d5c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:00.842 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.841+0000 7f0875ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:00.843 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.841+0000 7f0875ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:00.843 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.841+0000 7f0875ffb640 10 monclient: dump: 2024-03-25T19:40:00.843 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:00.843 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:00.843 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:00.843 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:00.843 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:00.843 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:00.843 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:00.843 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:00.843 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:00.843 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:00.843 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.841+0000 7f087d5c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:00.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.849+0000 7f0875ffb640 1 -- 172.21.15.44:0/3920010656 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f086c0017f0 con 0x7f087813f710 2024-03-25T19:40:00.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.849+0000 7f0875ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:00.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.849+0000 7f0875ffb640 1 -- 172.21.15.44:0/3920010656 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f086c001c00 con 0x7f087813f710 2024-03-25T19:40:00.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.849+0000 7f0875ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:00.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.849+0000 7f0875ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:00.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.849+0000 7f0875ffb640 10 monclient: dump: 2024-03-25T19:40:00.850 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:00.850 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:00.850 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:00.850 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:00.850 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:00.850 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:00.850 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:00.850 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:00.850 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:00.850 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:00.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.849+0000 7f087d5c6640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:00.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.849+0000 7f087d5c6640 4 set_mon_vals no callback set 2024-03-25T19:40:00.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.850+0000 7f087d5c6640 10 monclient: shutdown 2024-03-25T19:40:00.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.850+0000 7f087d5c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:00.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.850+0000 7f087d5c6640 1 -- 172.21.15.44:0/3920010656 >> v1:172.21.15.44:6790/0 conn(0x7f087813f710 legacy=0x7f08781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:00.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.850+0000 7f087d5c6640 1 -- 172.21.15.44:0/3920010656 shutdown_connections 2024-03-25T19:40:00.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.850+0000 7f087d5c6640 1 -- 172.21.15.44:0/3920010656 >> 172.21.15.44:0/3920010656 conn(0x7f08780b7ea0 msgr2=0x7f08780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:00.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.850+0000 7f087d5c6640 1 -- 172.21.15.44:0/3920010656 shutdown_connections 2024-03-25T19:40:00.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.850+0000 7f087d5c6640 1 -- 172.21.15.44:0/3920010656 wait complete. 2024-03-25T19:40:00.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.851+0000 7f087d5c6640 10 monclient: build_initial_monmap 2024-03-25T19:40:00.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.851+0000 7f087d5c6640 10 monclient: monmap: 2024-03-25T19:40:00.853 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:40:00.853 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:00.853 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:40:00.853 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:40:00.853 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:00.853 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:40:00.853 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:40:00.853 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:40:00.853 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:40:00.853 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:00.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.852+0000 7f087d5c6640 1 librados: starting msgr at 2024-03-25T19:40:00.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.852+0000 7f087d5c6640 1 librados: starting objecter 2024-03-25T19:40:00.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.852+0000 7f087d5c6640 1 Processor -- start 2024-03-25T19:40:00.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.852+0000 7f087d5c6640 1 -- start start 2024-03-25T19:40:00.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.852+0000 7f087d5c6640 1 librados: setting wanted keys 2024-03-25T19:40:00.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.852+0000 7f087d5c6640 1 librados: calling monclient init 2024-03-25T19:40:00.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.852+0000 7f087d5c6640 10 monclient: init 2024-03-25T19:40:00.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.853+0000 7f087d5c6640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:00.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.853+0000 7f087d5c6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:00.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.853+0000 7f087d5c6640 10 monclient(hunting): picked mon.noname-a con 0x7f08780a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:40:00.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.853+0000 7f087d5c6640 10 monclient(hunting): picked mon.noname-b con 0x7f087813f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:40:00.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.853+0000 7f087d5c6640 10 monclient(hunting): picked mon.noname-c con 0x7f08781430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:40:00.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.853+0000 7f087d5c6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08780abc80 con 0x7f08780a98f0 2024-03-25T19:40:00.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.853+0000 7f087d5c6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08781d7780 con 0x7f087813f710 2024-03-25T19:40:00.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.853+0000 7f087d5c6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08781399a0 con 0x7f08781430c0 2024-03-25T19:40:00.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.853+0000 7f087d5c6640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:00.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.853+0000 7f087d5c6640 10 monclient(hunting): authenticate will time out at 5665.024570s 2024-03-25T19:40:00.862 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.862+0000 7f08767fc640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f087813f710 0x7f08781d4840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:56070/0 (socket says 172.21.15.44:56070) 2024-03-25T19:40:00.862 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.862+0000 7f08767fc640 1 -- 172.21.15.44:0/3634221496 learned_addr learned my addr 172.21.15.44:0/3634221496 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:40:00.864 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.864+0000 7f085f7fe640 10 client.?.objecter ms_handle_connect 0x7f08781430c0 2024-03-25T19:40:00.864 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.864+0000 7f085f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:00.864 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.865+0000 7f085f7fe640 10 client.?.objecter ms_handle_connect 0x7f087813f710 2024-03-25T19:40:00.865 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.865+0000 7f085f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:00.873 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.873+0000 7f085f7fe640 1 -- 172.21.15.44:0/3634221496 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528680781 0 0) 0x7f08781d7780 con 0x7f087813f710 2024-03-25T19:40:00.873 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.873+0000 7f085f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:00.873 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.873+0000 7f085f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:00.873 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.873+0000 7f085f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:00.873 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.873+0000 7f085f7fe640 10 monclient(hunting): my global_id is 4426 2024-03-25T19:40:00.873 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.873+0000 7f085f7fe640 1 -- 172.21.15.44:0/3634221496 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0864004eb0 con 0x7f087813f710 2024-03-25T19:40:00.875 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.875+0000 7f085f7fe640 1 -- 172.21.15.44:0/3634221496 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738502804 0 0) 0x7f08781399a0 con 0x7f08781430c0 2024-03-25T19:40:00.875 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.875+0000 7f085f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:00.875 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.875+0000 7f085f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:00.875 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.875+0000 7f085f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:00.875 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.875+0000 7f085f7fe640 10 monclient(hunting): my global_id is 4415 2024-03-25T19:40:00.875 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.875+0000 7f085f7fe640 1 -- 172.21.15.44:0/3634221496 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08781d7780 con 0x7f08781430c0 2024-03-25T19:40:00.876 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.877+0000 7f085f7fe640 10 client.?.objecter ms_handle_connect 0x7f08780a98f0 2024-03-25T19:40:00.876 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.877+0000 7f085f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:00.892 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.893+0000 7f085f7fe640 1 -- 172.21.15.44:0/3634221496 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2491902390 0 0) 0x7f08780abc80 con 0x7f08780a98f0 2024-03-25T19:40:00.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.893+0000 7f085f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:00.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.893+0000 7f085f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:00.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.893+0000 7f085f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:00.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.893+0000 7f085f7fe640 10 monclient(hunting): my global_id is 4446 2024-03-25T19:40:00.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.893+0000 7f085f7fe640 1 -- 172.21.15.44:0/3634221496 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08781399a0 con 0x7f08780a98f0 2024-03-25T19:40:00.902 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.902+0000 7f085f7fe640 1 -- 172.21.15.44:0/3634221496 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4123385026 0 0) 0x7f0864004eb0 con 0x7f087813f710 2024-03-25T19:40:00.902 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.902+0000 7f085f7fe640 1 -- 172.21.15.44:0/3634221496 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08780abc80 con 0x7f087813f710 2024-03-25T19:40:00.902 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.902+0000 7f085f7fe640 1 -- 172.21.15.44:0/3634221496 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f086c001bc0 con 0x7f087813f710 2024-03-25T19:40:00.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.902+0000 7f085f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:00.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.902+0000 7f085f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:00.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.902+0000 7f085f7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:00.903 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:00.903 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:00.903 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:00.903 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:00.903 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:00.903 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:00.903 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:00.903 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:00.903 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:00.903 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:00.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.902+0000 7f085f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:00.916 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.916+0000 7f085f7fe640 1 -- 172.21.15.44:0/3634221496 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3991191127 0 0) 0x7f08781d7780 con 0x7f08781430c0 2024-03-25T19:40:00.916 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.916+0000 7f085f7fe640 1 -- 172.21.15.44:0/3634221496 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0864004eb0 con 0x7f08781430c0 2024-03-25T19:40:00.916 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.916+0000 7f085f7fe640 1 -- 172.21.15.44:0/3634221496 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f0870001bd0 con 0x7f08781430c0 2024-03-25T19:40:00.916 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.916+0000 7f085f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:00.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.916+0000 7f085f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:00.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.916+0000 7f085f7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:00.917 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:00.917 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:00.917 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:00.917 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:00.917 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:00.917 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:00.917 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:00.917 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:00.917 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:00.917 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:00.922 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.922+0000 7f085f7fe640 1 -- 172.21.15.44:0/3634221496 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1612935293 0 0) 0x7f08780abc80 con 0x7f087813f710 2024-03-25T19:40:00.923 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.922+0000 7f085f7fe640 1 -- 172.21.15.44:0/3634221496 >> v1:172.21.15.44:6790/0 conn(0x7f08781430c0 legacy=0x7f0878138600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:00.923 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.922+0000 7f085f7fe640 1 -- 172.21.15.44:0/3634221496 >> v1:172.21.15.44:6789/0 conn(0x7f08780a98f0 legacy=0x7f08781d7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:00.923 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.922+0000 7f085f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:00.923 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.922+0000 7f085f7fe640 1 monclient: found mon.b 2024-03-25T19:40:00.923 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.922+0000 7f085f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:00.923 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.922+0000 7f085f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:40:00.923 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.922+0000 7f085f7fe640 1 -- 172.21.15.44:0/3634221496 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08781d41e0 con 0x7f087813f710 2024-03-25T19:40:00.923 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.922+0000 7f085f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:00.923 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.922+0000 7f085f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:00.923 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.922+0000 7f085f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:00.923 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.922+0000 7f087d5c6640 5 monclient: authenticate success, global_id 4426 2024-03-25T19:40:00.923 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.922+0000 7f087d5c6640 10 monclient: _renew_subs 2024-03-25T19:40:00.924 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.922+0000 7f087d5c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:40:00.924 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.922+0000 7f087d5c6640 1 -- 172.21.15.44:0/3634221496 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08781d8370 con 0x7f087813f710 2024-03-25T19:40:00.924 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.922+0000 7f087d5c6640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:00.924 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.922+0000 7f087d5c6640 10 monclient: _renew_subs 2024-03-25T19:40:00.924 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.922+0000 7f087d5c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:40:00.924 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.922+0000 7f087d5c6640 1 -- 172.21.15.44:0/3634221496 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08781d5130 con 0x7f087813f710 2024-03-25T19:40:00.924 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.922+0000 7f087d5c6640 1 librados: init done 2024-03-25T19:40:00.924 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.923+0000 7f087d5c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:00.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.923+0000 7f087d5c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:00.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.923+0000 7f087d5c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:40:00.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:00.923+0000 7f087d5c6640 1 -- 172.21.15.44:0/3634221496 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08780627b0 con 0x7f087813f710 2024-03-25T19:40:01.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.212+0000 7f085f7fe640 1 -- 172.21.15.44:0/3634221496 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f086c0011f0 con 0x7f087813f710 2024-03-25T19:40:01.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.212+0000 7f085f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:01.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.212+0000 7f085f7fe640 1 -- 172.21.15.44:0/3634221496 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f086c003a40 con 0x7f087813f710 2024-03-25T19:40:01.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.212+0000 7f085f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:01.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.212+0000 7f085f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:01.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.212+0000 7f085f7fe640 10 monclient: dump: 2024-03-25T19:40:01.213 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:01.213 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:01.213 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:01.213 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:01.213 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:01.213 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:01.214 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:01.214 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:01.214 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:01.214 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:01.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.212+0000 7f08757fa640 4 set_mon_vals no callback set 2024-03-25T19:40:01.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.213+0000 7f085f7fe640 1 -- 172.21.15.44:0/3634221496 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f086c003c60 con 0x7f087813f710 2024-03-25T19:40:01.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.214+0000 7f0876ffd640 10 monclient: get_auth_request con 0x7f086406df20 auth_method 0 2024-03-25T19:40:01.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.214+0000 7f085f7fe640 10 client.4426.objecter ms_handle_connect 0x7f086406df20 2024-03-25T19:40:01.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.219+0000 7f085f7fe640 1 -- 172.21.15.44:0/3634221496 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f086c0548f0 con 0x7f087813f710 2024-03-25T19:40:01.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.219+0000 7f085f7fe640 10 client.4426.objecter ms_dispatch 0x7f0878000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:01.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.219+0000 7f085f7fe640 3 client.4426.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:01.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.219+0000 7f085f7fe640 3 client.4426.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:01.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.219+0000 7f085f7fe640 20 client.4426.objecter dump_active .. 0 homeless 2024-03-25T19:40:01.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.219+0000 7f085f7fe640 1 -- 172.21.15.44:0/3634221496 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f086c050230 con 0x7f087813f710 2024-03-25T19:40:01.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.219+0000 7f085f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:01.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.219+0000 7f085f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:01.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.471+0000 7f087d5c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:01.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.471+0000 7f087d5c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:01.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.471+0000 7f087d5c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:40:01.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.471+0000 7f087d5c6640 1 -- 172.21.15.44:0/3634221496 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f087813e2b0 con 0x7f087813f710 2024-03-25T19:40:01.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.508+0000 7f085f7fe640 1 -- 172.21.15.44:0/3634221496 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f086c052330 con 0x7f087813f710 2024-03-25T19:40:01.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.508+0000 7f085f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:01.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.508+0000 7f085f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:01.508 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:40:01.509 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:01.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.511+0000 7f087d5c6640 10 librados: watch_flush enter 2024-03-25T19:40:01.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.511+0000 7f087d5c6640 10 librados: watch_flush exit 2024-03-25T19:40:01.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.511+0000 7f087d5c6640 20 client.4426.objecter shutdown clearing up homeless session... 2024-03-25T19:40:01.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.511+0000 7f087d5c6640 10 client.4426.objecter successfully canceled tick 2024-03-25T19:40:01.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.511+0000 7f087d5c6640 1 -- 172.21.15.44:0/3634221496 >> v1:172.21.15.111:6800/3883215793 conn(0x7f086406df20 legacy=0x7f08640703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:01.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.511+0000 7f087d5c6640 10 monclient: shutdown 2024-03-25T19:40:01.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.511+0000 7f087d5c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:01.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.511+0000 7f087d5c6640 1 -- 172.21.15.44:0/3634221496 >> v1:172.21.15.111:6789/0 conn(0x7f087813f710 legacy=0x7f08781d4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:01.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.511+0000 7f087d5c6640 1 -- 172.21.15.44:0/3634221496 shutdown_connections 2024-03-25T19:40:01.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.511+0000 7f087d5c6640 1 -- 172.21.15.44:0/3634221496 >> 172.21.15.44:0/3634221496 conn(0x7f08780b7ea0 msgr2=0x7f0878074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:01.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.511+0000 7f087d5c6640 1 -- 172.21.15.44:0/3634221496 shutdown_connections 2024-03-25T19:40:01.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.511+0000 7f087d5c6640 1 -- 172.21.15.44:0/3634221496 wait complete. 2024-03-25T19:40:01.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:01.512+0000 7f087d5c6640 1 librados: shutdown 2024-03-25T19:40:01.523 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:08.328 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:08.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.444+0000 7f874eda8640 10 monclient: get_monmap_and_config 2024-03-25T19:40:08.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.444+0000 7f874eda8640 10 monclient: build_initial_monmap 2024-03-25T19:40:08.445 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.444+0000 7f874eda8640 10 monclient: monmap: 2024-03-25T19:40:08.445 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:40:08.445 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:08.445 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:40:08.445910+0000 2024-03-25T19:40:08.445 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:40:08.445910+0000 2024-03-25T19:40:08.445 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:08.445 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:08.445 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:40:08.445 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:40:08.445 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:40:08.445 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:08.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.445+0000 7f874eda8640 1 Processor -- start 2024-03-25T19:40:08.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.445+0000 7f874eda8640 1 -- start start 2024-03-25T19:40:08.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.445+0000 7f874eda8640 10 monclient: init 2024-03-25T19:40:08.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.446+0000 7f874eda8640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:08.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.446+0000 7f874eda8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:08.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.446+0000 7f874eda8640 10 monclient(hunting): picked mon.noname-c con 0x7f87480a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:40:08.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.446+0000 7f874eda8640 10 monclient(hunting): picked mon.noname-a con 0x7f874813f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:40:08.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.446+0000 7f874eda8640 10 monclient(hunting): picked mon.noname-b con 0x7f87481430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:40:08.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.446+0000 7f874eda8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8748142520 con 0x7f874813f710 2024-03-25T19:40:08.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.446+0000 7f874eda8640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87480b28a0 con 0x7f87480a98f0 2024-03-25T19:40:08.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.446+0000 7f874eda8640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87480b2a30 con 0x7f87481430c0 2024-03-25T19:40:08.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.446+0000 7f874eda8640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:08.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.446+0000 7f874eda8640 10 monclient(hunting): authenticate will time out at 5672.617217s 2024-03-25T19:40:08.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.455+0000 7f874d345640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f87481430c0 0x7f874814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:37014/0 (socket says 172.21.15.44:37014) 2024-03-25T19:40:08.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.455+0000 7f874d345640 1 -- 172.21.15.44:0/1820840611 learned_addr learned my addr 172.21.15.44:0/1820840611 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:40:08.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.472+0000 7f87477fe640 1 -- 172.21.15.44:0/1820840611 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245799570 0 0) 0x7f87480b2a30 con 0x7f87481430c0 2024-03-25T19:40:08.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.472+0000 7f87477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:08.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.472+0000 7f87477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:08.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.472+0000 7f87477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:08.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.472+0000 7f87477fe640 10 monclient(hunting): my global_id is 4418 2024-03-25T19:40:08.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.472+0000 7f87477fe640 1 -- 172.21.15.44:0/1820840611 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8738003520 con 0x7f87481430c0 2024-03-25T19:40:08.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.477+0000 7f87477fe640 1 -- 172.21.15.44:0/1820840611 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1512825022 0 0) 0x7f87480b28a0 con 0x7f87480a98f0 2024-03-25T19:40:08.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.477+0000 7f87477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:08.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.477+0000 7f87477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:08.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.477+0000 7f87477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:08.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.477+0000 7f87477fe640 10 monclient(hunting): my global_id is 4429 2024-03-25T19:40:08.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.477+0000 7f87477fe640 1 -- 172.21.15.44:0/1820840611 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87480b2a30 con 0x7f87480a98f0 2024-03-25T19:40:08.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.489+0000 7f87477fe640 1 -- 172.21.15.44:0/1820840611 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1642149804 0 0) 0x7f8738003520 con 0x7f87481430c0 2024-03-25T19:40:08.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.489+0000 7f87477fe640 1 -- 172.21.15.44:0/1820840611 >> v1:172.21.15.111:6789/0 conn(0x7f87480a98f0 legacy=0x7f87480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:08.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.489+0000 7f87477fe640 1 -- 172.21.15.44:0/1820840611 >> v1:172.21.15.44:6789/0 conn(0x7f874813f710 legacy=0x7f87481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:08.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.489+0000 7f87477fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:08.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.489+0000 7f87477fe640 1 monclient: found mon.noname-b 2024-03-25T19:40:08.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.489+0000 7f87477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:08.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.489+0000 7f87477fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:40:08.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.489+0000 7f87477fe640 1 -- 172.21.15.44:0/1820840611 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87480b44c0 con 0x7f87481430c0 2024-03-25T19:40:08.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.489+0000 7f87477fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:08.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.489+0000 7f87477fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:08.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.489+0000 7f87477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:08.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.489+0000 7f87477fe640 1 -- 172.21.15.44:0/1820840611 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f8740001a10 con 0x7f87481430c0 2024-03-25T19:40:08.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.489+0000 7f87477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:08.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.489+0000 7f87477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:08.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.489+0000 7f87477fe640 10 monclient: dump: 2024-03-25T19:40:08.491 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:08.491 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:08.491 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:08.491 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:08.491 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:08.491 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:08.491 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:08.491 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:08.491 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:08.491 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:08.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.489+0000 7f874eda8640 5 monclient: authenticate success, global_id 4418 2024-03-25T19:40:08.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.489+0000 7f874eda8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:08.499 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.499+0000 7f87477fe640 1 -- 172.21.15.44:0/1820840611 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8740001b70 con 0x7f87481430c0 2024-03-25T19:40:08.499 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.499+0000 7f87477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:08.499 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.499+0000 7f87477fe640 1 -- 172.21.15.44:0/1820840611 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f8740001f80 con 0x7f87481430c0 2024-03-25T19:40:08.499 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.499+0000 7f87477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:08.499 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.499+0000 7f87477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:08.499 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.499+0000 7f87477fe640 10 monclient: dump: 2024-03-25T19:40:08.499 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:08.499 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:08.499 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:08.500 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:08.500 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:08.500 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:08.500 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:08.500 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:08.500 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:08.500 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:08.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.499+0000 7f874eda8640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:08.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.499+0000 7f874eda8640 4 set_mon_vals no callback set 2024-03-25T19:40:08.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.500+0000 7f874eda8640 10 monclient: shutdown 2024-03-25T19:40:08.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.500+0000 7f874eda8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:08.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.500+0000 7f874eda8640 1 -- 172.21.15.44:0/1820840611 >> v1:172.21.15.44:6790/0 conn(0x7f87481430c0 legacy=0x7f874814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:08.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.500+0000 7f874eda8640 1 -- 172.21.15.44:0/1820840611 shutdown_connections 2024-03-25T19:40:08.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.500+0000 7f874eda8640 1 -- 172.21.15.44:0/1820840611 >> 172.21.15.44:0/1820840611 conn(0x7f87480b7ea0 msgr2=0x7f87480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:08.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.500+0000 7f874eda8640 1 -- 172.21.15.44:0/1820840611 shutdown_connections 2024-03-25T19:40:08.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.500+0000 7f874eda8640 1 -- 172.21.15.44:0/1820840611 wait complete. 2024-03-25T19:40:08.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.500+0000 7f874eda8640 10 monclient: build_initial_monmap 2024-03-25T19:40:08.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.500+0000 7f874eda8640 10 monclient: monmap: 2024-03-25T19:40:08.503 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:40:08.503 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:08.503 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:40:08.503 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:40:08.503 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:08.503 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:40:08.503 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:40:08.503 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:40:08.503 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:40:08.503 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:08.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.501+0000 7f874eda8640 1 librados: starting msgr at 2024-03-25T19:40:08.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.501+0000 7f874eda8640 1 librados: starting objecter 2024-03-25T19:40:08.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.501+0000 7f874eda8640 1 Processor -- start 2024-03-25T19:40:08.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.501+0000 7f874eda8640 1 -- start start 2024-03-25T19:40:08.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.501+0000 7f874eda8640 1 librados: setting wanted keys 2024-03-25T19:40:08.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.501+0000 7f874eda8640 1 librados: calling monclient init 2024-03-25T19:40:08.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.501+0000 7f874eda8640 10 monclient: init 2024-03-25T19:40:08.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.502+0000 7f874eda8640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:08.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.502+0000 7f874eda8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:08.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.502+0000 7f874eda8640 10 monclient(hunting): picked mon.noname-c con 0x7f87480a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T19:40:08.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.502+0000 7f874eda8640 10 monclient(hunting): picked mon.noname-b con 0x7f874813f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:40:08.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.502+0000 7f874eda8640 10 monclient(hunting): picked mon.noname-a con 0x7f87481430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:40:08.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.502+0000 7f874eda8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8748137ab0 con 0x7f87481430c0 2024-03-25T19:40:08.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.502+0000 7f874eda8640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8748138bd0 con 0x7f874813f710 2024-03-25T19:40:08.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.502+0000 7f874eda8640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8748138e40 con 0x7f87480a98f0 2024-03-25T19:40:08.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.502+0000 7f874eda8640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:08.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.502+0000 7f874eda8640 10 monclient(hunting): authenticate will time out at 5672.673150s 2024-03-25T19:40:08.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.510+0000 7f874cb44640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f87480a98f0 0x7f8748137f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:37060/0 (socket says 172.21.15.44:37060) 2024-03-25T19:40:08.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.510+0000 7f874cb44640 1 -- 172.21.15.44:0/2065839872 learned_addr learned my addr 172.21.15.44:0/2065839872 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:40:08.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.510+0000 7f87457fa640 10 client.?.objecter ms_handle_connect 0x7f87480a98f0 2024-03-25T19:40:08.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.510+0000 7f87457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:08.514 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.514+0000 7f87457fa640 10 client.?.objecter ms_handle_connect 0x7f87481430c0 2024-03-25T19:40:08.514 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.514+0000 7f87457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:08.514 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.515+0000 7f87457fa640 10 client.?.objecter ms_handle_connect 0x7f874813f710 2024-03-25T19:40:08.514 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.515+0000 7f87457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:08.528 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.529+0000 7f87457fa640 1 -- 172.21.15.44:0/2065839872 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856183096 0 0) 0x7f8748137ab0 con 0x7f87481430c0 2024-03-25T19:40:08.529 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.529+0000 7f87457fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:08.529 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.529+0000 7f87457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:08.529 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.529+0000 7f87457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:08.529 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.529+0000 7f87457fa640 10 monclient(hunting): my global_id is 4452 2024-03-25T19:40:08.529 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.529+0000 7f87457fa640 1 -- 172.21.15.44:0/2065839872 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f872c004eb0 con 0x7f87481430c0 2024-03-25T19:40:08.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.596+0000 7f87457fa640 1 -- 172.21.15.44:0/2065839872 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2446233494 0 0) 0x7f872c004eb0 con 0x7f87481430c0 2024-03-25T19:40:08.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.597+0000 7f87457fa640 1 -- 172.21.15.44:0/2065839872 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8748137ab0 con 0x7f87481430c0 2024-03-25T19:40:08.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.597+0000 7f87457fa640 1 -- 172.21.15.44:0/2065839872 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f8740001e80 con 0x7f87481430c0 2024-03-25T19:40:08.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.597+0000 7f87457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:08.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.597+0000 7f87457fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:08.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.597+0000 7f87457fa640 10 monclient(hunting): dump: 2024-03-25T19:40:08.597 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:08.597 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:08.597 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:08.597 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:08.597 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:08.597 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:08.597 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:08.597 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:08.598 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:08.598 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:08.598 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.597+0000 7f87457fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:08.605 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.605+0000 7f87457fa640 1 -- 172.21.15.44:0/2065839872 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2222333139 0 0) 0x7f8748137ab0 con 0x7f87481430c0 2024-03-25T19:40:08.605 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.605+0000 7f87457fa640 1 -- 172.21.15.44:0/2065839872 >> v1:172.21.15.44:6790/0 conn(0x7f87480a98f0 legacy=0x7f8748137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:08.605 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.605+0000 7f87457fa640 1 -- 172.21.15.44:0/2065839872 >> v1:172.21.15.111:6789/0 conn(0x7f874813f710 legacy=0x7f87481df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:08.605 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.605+0000 7f87457fa640 10 monclient: _finish_hunting 0 2024-03-25T19:40:08.605 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.605+0000 7f87457fa640 1 monclient: found mon.a 2024-03-25T19:40:08.605 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.605+0000 7f87457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:08.605 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.605+0000 7f87457fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:40:08.605 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.605+0000 7f87457fa640 1 -- 172.21.15.44:0/2065839872 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8748139120 con 0x7f87481430c0 2024-03-25T19:40:08.605 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.605+0000 7f87457fa640 10 monclient: _finish_auth 0 2024-03-25T19:40:08.605 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.605+0000 7f87457fa640 10 monclient: _check_auth_tickets 2024-03-25T19:40:08.605 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.605+0000 7f87457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:08.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.605+0000 7f874eda8640 5 monclient: authenticate success, global_id 4452 2024-03-25T19:40:08.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.605+0000 7f874eda8640 10 monclient: _renew_subs 2024-03-25T19:40:08.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.605+0000 7f874eda8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:40:08.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.605+0000 7f874eda8640 1 -- 172.21.15.44:0/2065839872 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8748139400 con 0x7f87481430c0 2024-03-25T19:40:08.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.605+0000 7f874eda8640 10 client.4452.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:08.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.605+0000 7f874eda8640 10 monclient: _renew_subs 2024-03-25T19:40:08.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.605+0000 7f874eda8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:40:08.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.605+0000 7f874eda8640 1 -- 172.21.15.44:0/2065839872 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87480b7880 con 0x7f87481430c0 2024-03-25T19:40:08.606 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.605+0000 7f874eda8640 1 librados: init done 2024-03-25T19:40:08.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.606+0000 7f874eda8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:08.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.606+0000 7f874eda8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:08.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.606+0000 7f874eda8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:40:08.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.606+0000 7f874eda8640 1 -- 172.21.15.44:0/2065839872 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8748137810 con 0x7f87481430c0 2024-03-25T19:40:08.617 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.617+0000 7f87457fa640 1 -- 172.21.15.44:0/2065839872 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8740002080 con 0x7f87481430c0 2024-03-25T19:40:08.617 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.618+0000 7f87457fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:08.617 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.618+0000 7f87457fa640 1 -- 172.21.15.44:0/2065839872 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f8740004fb0 con 0x7f87481430c0 2024-03-25T19:40:08.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.618+0000 7f87457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:08.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.618+0000 7f87457fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:08.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.618+0000 7f87457fa640 10 monclient: dump: 2024-03-25T19:40:08.618 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:08.618 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:08.618 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:08.618 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:08.618 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:08.618 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:08.618 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:08.618 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:08.618 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:08.618 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:08.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.618+0000 7f8746ffd640 4 set_mon_vals no callback set 2024-03-25T19:40:08.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.635+0000 7f87457fa640 1 -- 172.21.15.44:0/2065839872 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f874001af80 con 0x7f87481430c0 2024-03-25T19:40:08.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.635+0000 7f874cb44640 10 monclient: get_auth_request con 0x7f872c06dc70 auth_method 0 2024-03-25T19:40:08.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.636+0000 7f87457fa640 10 client.4452.objecter ms_handle_connect 0x7f872c06dc70 2024-03-25T19:40:08.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.890+0000 7f87457fa640 1 -- 172.21.15.44:0/2065839872 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f8740054b20 con 0x7f87481430c0 2024-03-25T19:40:08.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.890+0000 7f87457fa640 10 client.4452.objecter ms_dispatch 0x7f8748000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:08.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.890+0000 7f87457fa640 3 client.4452.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:08.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.890+0000 7f87457fa640 3 client.4452.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:08.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.890+0000 7f87457fa640 20 client.4452.objecter dump_active .. 0 homeless 2024-03-25T19:40:08.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.890+0000 7f87457fa640 1 -- 172.21.15.44:0/2065839872 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8740050460 con 0x7f87481430c0 2024-03-25T19:40:08.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.890+0000 7f87457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:08.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:08.890+0000 7f87457fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:09.124 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:09.124+0000 7f874eda8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:09.124 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:09.124+0000 7f874eda8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:09.124 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:09.124+0000 7f874eda8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:40:09.124 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:09.124+0000 7f874eda8640 1 -- 172.21.15.44:0/2065839872 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87480627b0 con 0x7f87481430c0 2024-03-25T19:40:09.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:09.155+0000 7f87457fa640 1 -- 172.21.15.44:0/2065839872 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f8740052560 con 0x7f87481430c0 2024-03-25T19:40:09.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:09.155+0000 7f87457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:09.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:09.155+0000 7f87457fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:09.156 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:40:09.156 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:09.158 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:09.158+0000 7f874eda8640 10 librados: watch_flush enter 2024-03-25T19:40:09.159 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:09.158+0000 7f874eda8640 10 librados: watch_flush exit 2024-03-25T19:40:09.159 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:09.158+0000 7f874eda8640 20 client.4452.objecter shutdown clearing up homeless session... 2024-03-25T19:40:09.159 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:09.158+0000 7f874eda8640 10 client.4452.objecter successfully canceled tick 2024-03-25T19:40:09.159 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:09.158+0000 7f874eda8640 1 -- 172.21.15.44:0/2065839872 >> v1:172.21.15.111:6800/3883215793 conn(0x7f872c06dc70 legacy=0x7f872c070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:09.159 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:09.158+0000 7f874eda8640 10 monclient: shutdown 2024-03-25T19:40:09.159 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:09.158+0000 7f874eda8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:09.159 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:09.158+0000 7f874eda8640 1 -- 172.21.15.44:0/2065839872 >> v1:172.21.15.44:6789/0 conn(0x7f87481430c0 legacy=0x7f87481e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:09.159 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:09.158+0000 7f874eda8640 1 -- 172.21.15.44:0/2065839872 shutdown_connections 2024-03-25T19:40:09.159 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:09.158+0000 7f874eda8640 1 -- 172.21.15.44:0/2065839872 >> 172.21.15.44:0/2065839872 conn(0x7f87480b7ea0 msgr2=0x7f8748074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:09.159 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:09.158+0000 7f874eda8640 1 -- 172.21.15.44:0/2065839872 shutdown_connections 2024-03-25T19:40:09.159 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:09.158+0000 7f874eda8640 1 -- 172.21.15.44:0/2065839872 wait complete. 2024-03-25T19:40:09.159 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:09.158+0000 7f874eda8640 1 librados: shutdown 2024-03-25T19:40:09.170 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:15.977 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:16.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.094+0000 7f37db6dd640 10 monclient: get_monmap_and_config 2024-03-25T19:40:16.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.094+0000 7f37db6dd640 10 monclient: build_initial_monmap 2024-03-25T19:40:16.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.094+0000 7f37db6dd640 10 monclient: monmap: 2024-03-25T19:40:16.094 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:40:16.094 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:16.094 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:40:16.095473+0000 2024-03-25T19:40:16.094 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:40:16.095473+0000 2024-03-25T19:40:16.094 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:16.094 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:16.095 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:40:16.095 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:40:16.095 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:40:16.095 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:16.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.095+0000 7f37db6dd640 1 Processor -- start 2024-03-25T19:40:16.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.095+0000 7f37db6dd640 1 -- start start 2024-03-25T19:40:16.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.095+0000 7f37db6dd640 10 monclient: init 2024-03-25T19:40:16.096 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.096+0000 7f37db6dd640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:16.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.096+0000 7f37db6dd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:16.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.096+0000 7f37db6dd640 10 monclient(hunting): picked mon.noname-c con 0x7f37d4149f50 addr v1:172.21.15.111:6789/0 2024-03-25T19:40:16.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.096+0000 7f37db6dd640 10 monclient(hunting): picked mon.noname-b con 0x7f37d406c780 addr v1:172.21.15.44:6790/0 2024-03-25T19:40:16.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.096+0000 7f37db6dd640 10 monclient(hunting): picked mon.noname-a con 0x7f37d40b1b10 addr v1:172.21.15.44:6789/0 2024-03-25T19:40:16.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.096+0000 7f37db6dd640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37d414a330 con 0x7f37d40b1b10 2024-03-25T19:40:16.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.096+0000 7f37db6dd640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37d4070660 con 0x7f37d4149f50 2024-03-25T19:40:16.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.096+0000 7f37db6dd640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37d406d7c0 con 0x7f37d406c780 2024-03-25T19:40:16.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.096+0000 7f37db6dd640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:16.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.096+0000 7f37db6dd640 10 monclient(hunting): authenticate will time out at 5680.267335s 2024-03-25T19:40:16.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.104+0000 7f37d9479640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f37d4149f50 0x7f37d40b0710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:43896/0 (socket says 172.21.15.44:43896) 2024-03-25T19:40:16.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.104+0000 7f37d9479640 1 -- 172.21.15.44:0/1808200269 learned_addr learned my addr 172.21.15.44:0/1808200269 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:40:16.127 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.127+0000 7f37cbfff640 1 -- 172.21.15.44:0/1808200269 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800941970 0 0) 0x7f37d414a330 con 0x7f37d40b1b10 2024-03-25T19:40:16.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.128+0000 7f37cbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:16.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.128+0000 7f37cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:16.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.128+0000 7f37cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:16.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.128+0000 7f37cbfff640 10 monclient(hunting): my global_id is 4455 2024-03-25T19:40:16.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.128+0000 7f37cbfff640 1 -- 172.21.15.44:0/1808200269 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37c0003520 con 0x7f37d40b1b10 2024-03-25T19:40:16.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.131+0000 7f37cbfff640 1 -- 172.21.15.44:0/1808200269 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 828743771 0 0) 0x7f37d406d7c0 con 0x7f37d406c780 2024-03-25T19:40:16.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.131+0000 7f37cbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:16.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.131+0000 7f37cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:16.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.131+0000 7f37cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:16.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.131+0000 7f37cbfff640 10 monclient(hunting): my global_id is 4424 2024-03-25T19:40:16.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.131+0000 7f37cbfff640 1 -- 172.21.15.44:0/1808200269 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37d414a330 con 0x7f37d406c780 2024-03-25T19:40:16.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.134+0000 7f37cbfff640 1 -- 172.21.15.44:0/1808200269 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381902199 0 0) 0x7f37d4070660 con 0x7f37d4149f50 2024-03-25T19:40:16.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.134+0000 7f37cbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:16.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.134+0000 7f37cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:16.135 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.134+0000 7f37cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:16.135 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.134+0000 7f37cbfff640 10 monclient(hunting): my global_id is 4435 2024-03-25T19:40:16.135 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.134+0000 7f37cbfff640 1 -- 172.21.15.44:0/1808200269 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37d406d7c0 con 0x7f37d4149f50 2024-03-25T19:40:16.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.139+0000 7f37cbfff640 1 -- 172.21.15.44:0/1808200269 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2874010495 0 0) 0x7f37c0003520 con 0x7f37d40b1b10 2024-03-25T19:40:16.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.139+0000 7f37cbfff640 1 -- 172.21.15.44:0/1808200269 >> v1:172.21.15.44:6790/0 conn(0x7f37d406c780 legacy=0x7f37d40b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:16.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.139+0000 7f37cbfff640 1 -- 172.21.15.44:0/1808200269 >> v1:172.21.15.111:6789/0 conn(0x7f37d4149f50 legacy=0x7f37d40b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:16.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.139+0000 7f37cbfff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:16.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.139+0000 7f37cbfff640 1 monclient: found mon.noname-a 2024-03-25T19:40:16.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.139+0000 7f37cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:16.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.139+0000 7f37cbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:40:16.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.139+0000 7f37cbfff640 1 -- 172.21.15.44:0/1808200269 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d406d3e0 con 0x7f37d40b1b10 2024-03-25T19:40:16.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.139+0000 7f37cbfff640 10 monclient: _finish_auth 0 2024-03-25T19:40:16.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.139+0000 7f37cbfff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:16.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.139+0000 7f37cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:16.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.139+0000 7f37cbfff640 1 -- 172.21.15.44:0/1808200269 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f37cc001a10 con 0x7f37d40b1b10 2024-03-25T19:40:16.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.139+0000 7f37cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:16.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.139+0000 7f37cbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:16.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.139+0000 7f37cbfff640 10 monclient: dump: 2024-03-25T19:40:16.141 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:16.141 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:16.141 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:16.141 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:16.141 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:16.141 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:16.141 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:16.141 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:16.141 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:16.141 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:16.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.139+0000 7f37db6dd640 5 monclient: authenticate success, global_id 4455 2024-03-25T19:40:16.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.139+0000 7f37db6dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:16.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.147+0000 7f37cbfff640 1 -- 172.21.15.44:0/1808200269 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37cc001b70 con 0x7f37d40b1b10 2024-03-25T19:40:16.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.147+0000 7f37cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:16.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.147+0000 7f37cbfff640 1 -- 172.21.15.44:0/1808200269 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f37cc001f80 con 0x7f37d40b1b10 2024-03-25T19:40:16.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.147+0000 7f37cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:16.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.147+0000 7f37cbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:16.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.147+0000 7f37cbfff640 10 monclient: dump: 2024-03-25T19:40:16.148 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:16.148 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:16.148 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:16.148 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:16.148 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:16.148 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:16.149 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:16.149 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:16.149 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:16.149 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:16.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.148+0000 7f37db6dd640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:16.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.148+0000 7f37db6dd640 4 set_mon_vals no callback set 2024-03-25T19:40:16.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.149+0000 7f37db6dd640 10 monclient: shutdown 2024-03-25T19:40:16.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.149+0000 7f37db6dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:16.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.149+0000 7f37db6dd640 1 -- 172.21.15.44:0/1808200269 >> v1:172.21.15.44:6789/0 conn(0x7f37d40b1b10 legacy=0x7f37d4152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:16.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.149+0000 7f37db6dd640 1 -- 172.21.15.44:0/1808200269 shutdown_connections 2024-03-25T19:40:16.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.149+0000 7f37db6dd640 1 -- 172.21.15.44:0/1808200269 >> 172.21.15.44:0/1808200269 conn(0x7f37d40b5d10 msgr2=0x7f37d40b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:16.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.149+0000 7f37db6dd640 1 -- 172.21.15.44:0/1808200269 shutdown_connections 2024-03-25T19:40:16.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.149+0000 7f37db6dd640 1 -- 172.21.15.44:0/1808200269 wait complete. 2024-03-25T19:40:16.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.150+0000 7f37db6dd640 10 monclient: build_initial_monmap 2024-03-25T19:40:16.151 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.150+0000 7f37db6dd640 10 monclient: monmap: 2024-03-25T19:40:16.151 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:40:16.152 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:16.152 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:40:16.152 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:40:16.152 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:16.152 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:40:16.152 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:40:16.152 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:40:16.152 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:40:16.152 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:16.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.150+0000 7f37db6dd640 1 librados: starting msgr at 2024-03-25T19:40:16.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.150+0000 7f37db6dd640 1 librados: starting objecter 2024-03-25T19:40:16.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.150+0000 7f37db6dd640 1 Processor -- start 2024-03-25T19:40:16.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.150+0000 7f37db6dd640 1 -- start start 2024-03-25T19:40:16.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.150+0000 7f37db6dd640 1 librados: setting wanted keys 2024-03-25T19:40:16.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.150+0000 7f37db6dd640 1 librados: calling monclient init 2024-03-25T19:40:16.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.150+0000 7f37db6dd640 10 monclient: init 2024-03-25T19:40:16.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.151+0000 7f37db6dd640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:16.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.151+0000 7f37db6dd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:16.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.151+0000 7f37db6dd640 10 monclient(hunting): picked mon.noname-a con 0x7f37d406c780 addr v1:172.21.15.44:6789/0 2024-03-25T19:40:16.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.151+0000 7f37db6dd640 10 monclient(hunting): picked mon.noname-c con 0x7f37d40b1b10 addr v1:172.21.15.44:6790/0 2024-03-25T19:40:16.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.151+0000 7f37db6dd640 10 monclient(hunting): picked mon.noname-b con 0x7f37d4149f50 addr v1:172.21.15.111:6789/0 2024-03-25T19:40:16.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.151+0000 7f37db6dd640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37d41dc5c0 con 0x7f37d406c780 2024-03-25T19:40:16.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.151+0000 7f37db6dd640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37d40a9b80 con 0x7f37d4149f50 2024-03-25T19:40:16.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.151+0000 7f37db6dd640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37d40afa70 con 0x7f37d40b1b10 2024-03-25T19:40:16.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.151+0000 7f37db6dd640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:16.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.151+0000 7f37db6dd640 10 monclient(hunting): authenticate will time out at 5680.321992s 2024-03-25T19:40:16.160 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.160+0000 7f37d8c78640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f37d40b1b10 0x7f37d40aed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:59506/0 (socket says 172.21.15.44:59506) 2024-03-25T19:40:16.160 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.160+0000 7f37d8c78640 1 -- 172.21.15.44:0/2735251195 learned_addr learned my addr 172.21.15.44:0/2735251195 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:40:16.163 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.163+0000 7f37c9ffb640 10 client.?.objecter ms_handle_connect 0x7f37d40b1b10 2024-03-25T19:40:16.163 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.163+0000 7f37c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:16.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.166+0000 7f37c9ffb640 10 client.?.objecter ms_handle_connect 0x7f37d4149f50 2024-03-25T19:40:16.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.166+0000 7f37c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:16.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.174+0000 7f37c9ffb640 1 -- 172.21.15.44:0/2735251195 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743951293 0 0) 0x7f37d40afa70 con 0x7f37d40b1b10 2024-03-25T19:40:16.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.174+0000 7f37c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:16.174 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.174+0000 7f37c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:16.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.174+0000 7f37c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:16.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.174+0000 7f37c9ffb640 10 monclient(hunting): my global_id is 4427 2024-03-25T19:40:16.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.174+0000 7f37c9ffb640 1 -- 172.21.15.44:0/2735251195 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37b8004eb0 con 0x7f37d40b1b10 2024-03-25T19:40:16.176 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.176+0000 7f37c9ffb640 10 client.?.objecter ms_handle_connect 0x7f37d406c780 2024-03-25T19:40:16.176 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.176+0000 7f37c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:16.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.181+0000 7f37c9ffb640 1 -- 172.21.15.44:0/2735251195 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860032239 0 0) 0x7f37d40a9b80 con 0x7f37d4149f50 2024-03-25T19:40:16.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.181+0000 7f37c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:16.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.181+0000 7f37c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:16.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.181+0000 7f37c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:16.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.181+0000 7f37c9ffb640 10 monclient(hunting): my global_id is 4438 2024-03-25T19:40:16.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.181+0000 7f37c9ffb640 1 -- 172.21.15.44:0/2735251195 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37d40afa70 con 0x7f37d4149f50 2024-03-25T19:40:16.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.196+0000 7f37c9ffb640 1 -- 172.21.15.44:0/2735251195 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2734384061 0 0) 0x7f37d41dc5c0 con 0x7f37d406c780 2024-03-25T19:40:16.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.196+0000 7f37c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:16.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.196+0000 7f37c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:16.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.196+0000 7f37c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:16.196 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.196+0000 7f37c9ffb640 10 monclient(hunting): my global_id is 4458 2024-03-25T19:40:16.197 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.196+0000 7f37c9ffb640 1 -- 172.21.15.44:0/2735251195 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37d40a9b80 con 0x7f37d406c780 2024-03-25T19:40:16.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.204+0000 7f37c9ffb640 1 -- 172.21.15.44:0/2735251195 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1614996517 0 0) 0x7f37b8004eb0 con 0x7f37d40b1b10 2024-03-25T19:40:16.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.205+0000 7f37c9ffb640 1 -- 172.21.15.44:0/2735251195 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37d41dc5c0 con 0x7f37d40b1b10 2024-03-25T19:40:16.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.205+0000 7f37c9ffb640 1 -- 172.21.15.44:0/2735251195 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f37c4001a30 con 0x7f37d40b1b10 2024-03-25T19:40:16.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.205+0000 7f37c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:16.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.205+0000 7f37c9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:16.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.205+0000 7f37c9ffb640 10 monclient(hunting): dump: 2024-03-25T19:40:16.207 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:16.207 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:16.207 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:16.207 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:16.207 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:16.207 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:16.207 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:16.207 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:16.208 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:16.208 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:16.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.205+0000 7f37c9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:16.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.205+0000 7f37c9ffb640 1 -- 172.21.15.44:0/2735251195 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3750820278 0 0) 0x7f37d40a9b80 con 0x7f37d406c780 2024-03-25T19:40:16.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.205+0000 7f37c9ffb640 1 -- 172.21.15.44:0/2735251195 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37b8004eb0 con 0x7f37d406c780 2024-03-25T19:40:16.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.205+0000 7f37c9ffb640 1 -- 172.21.15.44:0/2735251195 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f37d0001790 con 0x7f37d406c780 2024-03-25T19:40:16.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.205+0000 7f37c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:16.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.205+0000 7f37c9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:16.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.205+0000 7f37c9ffb640 10 monclient(hunting): dump: 2024-03-25T19:40:16.208 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:16.208 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:16.208 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:16.208 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:16.208 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:16.209 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:16.209 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:16.209 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:16.209 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:16.209 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:16.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.207+0000 7f37c9ffb640 1 -- 172.21.15.44:0/2735251195 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 567453028 0 0) 0x7f37d40afa70 con 0x7f37d4149f50 2024-03-25T19:40:16.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.207+0000 7f37c9ffb640 1 -- 172.21.15.44:0/2735251195 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37d40a9b80 con 0x7f37d4149f50 2024-03-25T19:40:16.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.207+0000 7f37c9ffb640 1 -- 172.21.15.44:0/2735251195 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f37cc001e80 con 0x7f37d4149f50 2024-03-25T19:40:16.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.207+0000 7f37c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:16.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.207+0000 7f37c9ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:16.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.207+0000 7f37c9ffb640 10 monclient(hunting): dump: 2024-03-25T19:40:16.210 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:16.210 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:16.210 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:16.210 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:16.210 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:16.210 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:16.210 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:16.210 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:16.210 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:16.210 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:16.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.219+0000 7f37c9ffb640 1 -- 172.21.15.44:0/2735251195 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1783631755 0 0) 0x7f37b8004eb0 con 0x7f37d406c780 2024-03-25T19:40:16.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.219+0000 7f37c9ffb640 1 -- 172.21.15.44:0/2735251195 >> v1:172.21.15.44:6790/0 conn(0x7f37d40b1b10 legacy=0x7f37d40aed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:16.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.219+0000 7f37c9ffb640 1 -- 172.21.15.44:0/2735251195 >> v1:172.21.15.111:6789/0 conn(0x7f37d4149f50 legacy=0x7f37d41e7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:16.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.219+0000 7f37c9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:40:16.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.219+0000 7f37c9ffb640 1 monclient: found mon.a 2024-03-25T19:40:16.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.219+0000 7f37c9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:16.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.219+0000 7f37c9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:40:16.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.219+0000 7f37c9ffb640 1 -- 172.21.15.44:0/2735251195 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d40afc00 con 0x7f37d406c780 2024-03-25T19:40:16.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.219+0000 7f37c9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:40:16.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.219+0000 7f37c9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:40:16.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.219+0000 7f37c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:16.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.219+0000 7f37db6dd640 5 monclient: authenticate success, global_id 4458 2024-03-25T19:40:16.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.219+0000 7f37db6dd640 10 monclient: _renew_subs 2024-03-25T19:40:16.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.219+0000 7f37db6dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:40:16.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.219+0000 7f37db6dd640 1 -- 172.21.15.44:0/2735251195 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37d40afdf0 con 0x7f37d406c780 2024-03-25T19:40:16.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.220+0000 7f37db6dd640 10 client.4458.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:16.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.220+0000 7f37db6dd640 10 monclient: _renew_subs 2024-03-25T19:40:16.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.220+0000 7f37db6dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:40:16.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.220+0000 7f37db6dd640 1 -- 172.21.15.44:0/2735251195 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37d4148900 con 0x7f37d406c780 2024-03-25T19:40:16.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.220+0000 7f37db6dd640 1 librados: init done 2024-03-25T19:40:16.222 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.221+0000 7f37db6dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:16.222 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.221+0000 7f37db6dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:16.222 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.221+0000 7f37db6dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:40:16.222 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.221+0000 7f37db6dd640 1 -- 172.21.15.44:0/2735251195 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37d40680d0 con 0x7f37d406c780 2024-03-25T19:40:16.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.537+0000 7f37c9ffb640 1 -- 172.21.15.44:0/2735251195 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37d0001ca0 con 0x7f37d406c780 2024-03-25T19:40:16.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.537+0000 7f37c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:16.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.537+0000 7f37c9ffb640 1 -- 172.21.15.44:0/2735251195 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f37d0004980 con 0x7f37d406c780 2024-03-25T19:40:16.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.537+0000 7f37c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:16.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.537+0000 7f37c9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:16.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.537+0000 7f37c9ffb640 10 monclient: dump: 2024-03-25T19:40:16.538 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:16.538 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:16.538 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:16.538 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:16.538 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:16.539 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:16.539 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:16.539 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:16.539 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:16.539 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:16.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.537+0000 7f37cb7fe640 4 set_mon_vals no callback set 2024-03-25T19:40:16.539 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.538+0000 7f37c9ffb640 1 -- 172.21.15.44:0/2735251195 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f37d001a990 con 0x7f37d406c780 2024-03-25T19:40:16.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.539+0000 7f37d8c78640 10 monclient: get_auth_request con 0x7f37b806e110 auth_method 0 2024-03-25T19:40:16.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.539+0000 7f37c9ffb640 10 client.4458.objecter ms_handle_connect 0x7f37b806e110 2024-03-25T19:40:16.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.545+0000 7f37c9ffb640 1 -- 172.21.15.44:0/2735251195 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f37d0054690 con 0x7f37d406c780 2024-03-25T19:40:16.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.545+0000 7f37c9ffb640 10 client.4458.objecter ms_dispatch 0x7f37d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:16.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.545+0000 7f37c9ffb640 3 client.4458.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:16.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.545+0000 7f37c9ffb640 3 client.4458.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:16.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.545+0000 7f37c9ffb640 20 client.4458.objecter dump_active .. 0 homeless 2024-03-25T19:40:16.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.545+0000 7f37c9ffb640 1 -- 172.21.15.44:0/2735251195 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f37d004ffd0 con 0x7f37d406c780 2024-03-25T19:40:16.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.545+0000 7f37c9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:16.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.545+0000 7f37c9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:16.764 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.764+0000 7f37db6dd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:16.764 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.764+0000 7f37db6dd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:16.764 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.764+0000 7f37db6dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:40:16.764 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.764+0000 7f37db6dd640 1 -- 172.21.15.44:0/2735251195 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37d40627b0 con 0x7f37d406c780 2024-03-25T19:40:16.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.787+0000 7f37c9ffb640 1 -- 172.21.15.44:0/2735251195 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f37d00520d0 con 0x7f37d406c780 2024-03-25T19:40:16.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.787+0000 7f37c9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:16.788 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.787+0000 7f37c9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:16.788 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:40:16.788 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:16.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.789+0000 7f37db6dd640 10 librados: watch_flush enter 2024-03-25T19:40:16.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.789+0000 7f37db6dd640 10 librados: watch_flush exit 2024-03-25T19:40:16.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.789+0000 7f37db6dd640 20 client.4458.objecter shutdown clearing up homeless session... 2024-03-25T19:40:16.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.789+0000 7f37db6dd640 10 client.4458.objecter successfully canceled tick 2024-03-25T19:40:16.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.789+0000 7f37db6dd640 1 -- 172.21.15.44:0/2735251195 >> v1:172.21.15.111:6800/3883215793 conn(0x7f37b806e110 legacy=0x7f37b80705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:16.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.789+0000 7f37db6dd640 10 monclient: shutdown 2024-03-25T19:40:16.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.789+0000 7f37db6dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:16.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.789+0000 7f37db6dd640 1 -- 172.21.15.44:0/2735251195 >> v1:172.21.15.44:6789/0 conn(0x7f37d406c780 legacy=0x7f37d40a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:16.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.789+0000 7f37db6dd640 1 -- 172.21.15.44:0/2735251195 shutdown_connections 2024-03-25T19:40:16.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.789+0000 7f37db6dd640 1 -- 172.21.15.44:0/2735251195 >> 172.21.15.44:0/2735251195 conn(0x7f37d40b5d10 msgr2=0x7f37d40b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:16.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.790+0000 7f37db6dd640 1 -- 172.21.15.44:0/2735251195 shutdown_connections 2024-03-25T19:40:16.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.790+0000 7f37db6dd640 1 -- 172.21.15.44:0/2735251195 wait complete. 2024-03-25T19:40:16.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:16.790+0000 7f37db6dd640 1 librados: shutdown 2024-03-25T19:40:16.800 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:19.491 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:05:10.393 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:19.505 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:05:10.406 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:19.519 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:05:10.399 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:19.529 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:05:10.431 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:19.554 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:05:10.456 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:19.561 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:05:10.463 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:19.586 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:05:10.465 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:19.596 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:05:10.498 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:19.613 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:05:10.493 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:23.607 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:23.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.722+0000 7f8623e43640 10 monclient: get_monmap_and_config 2024-03-25T19:40:23.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.722+0000 7f8623e43640 10 monclient: build_initial_monmap 2024-03-25T19:40:23.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.722+0000 7f8623e43640 10 monclient: monmap: 2024-03-25T19:40:23.724 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:40:23.724 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:23.724 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:40:23.724041+0000 2024-03-25T19:40:23.724 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:40:23.724041+0000 2024-03-25T19:40:23.724 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:23.724 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:23.725 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:40:23.725 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:40:23.725 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:40:23.725 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:23.725 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.723+0000 7f8623e43640 1 Processor -- start 2024-03-25T19:40:23.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.723+0000 7f8623e43640 1 -- start start 2024-03-25T19:40:23.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.723+0000 7f8623e43640 10 monclient: init 2024-03-25T19:40:23.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.724+0000 7f8623e43640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:23.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.724+0000 7f8623e43640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:23.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.724+0000 7f8623e43640 10 monclient(hunting): picked mon.noname-c con 0x7f861c0b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T19:40:23.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.724+0000 7f8623e43640 10 monclient(hunting): picked mon.noname-b con 0x7f861c0b1730 addr v1:172.21.15.44:6790/0 2024-03-25T19:40:23.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.724+0000 7f8623e43640 10 monclient(hunting): picked mon.noname-a con 0x7f861c14be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:40:23.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.724+0000 7f8623e43640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861c0af940 con 0x7f861c14be90 2024-03-25T19:40:23.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.724+0000 7f8623e43640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861c0b0a70 con 0x7f861c0b4d60 2024-03-25T19:40:23.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.724+0000 7f8623e43640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861c0b0d50 con 0x7f861c0b1730 2024-03-25T19:40:23.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.724+0000 7f8623e43640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:23.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.724+0000 7f8623e43640 10 monclient(hunting): authenticate will time out at 5687.895407s 2024-03-25T19:40:23.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.729+0000 7f8621bdf640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f861c0b4d60 0x7f861c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:46400/0 (socket says 172.21.15.44:46400) 2024-03-25T19:40:23.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.729+0000 7f8621bdf640 1 -- 172.21.15.44:0/3525945241 learned_addr learned my addr 172.21.15.44:0/3525945241 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:40:23.746 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.746+0000 7f8620bdd640 1 -- 172.21.15.44:0/3525945241 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2590547322 0 0) 0x7f861c0b0d50 con 0x7f861c0b1730 2024-03-25T19:40:23.746 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.746+0000 7f8620bdd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:23.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.746+0000 7f8620bdd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:23.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.746+0000 7f8620bdd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:23.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.746+0000 7f8620bdd640 10 monclient(hunting): my global_id is 4430 2024-03-25T19:40:23.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.746+0000 7f8620bdd640 1 -- 172.21.15.44:0/3525945241 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8608003520 con 0x7f861c0b1730 2024-03-25T19:40:23.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.746+0000 7f8620bdd640 1 -- 172.21.15.44:0/3525945241 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744976102 0 0) 0x7f861c0b0a70 con 0x7f861c0b4d60 2024-03-25T19:40:23.748 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.746+0000 7f8620bdd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:23.748 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.746+0000 7f8620bdd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:23.748 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.746+0000 7f8620bdd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:23.748 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.746+0000 7f8620bdd640 10 monclient(hunting): my global_id is 4441 2024-03-25T19:40:23.748 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.746+0000 7f8620bdd640 1 -- 172.21.15.44:0/3525945241 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f861c0b0d50 con 0x7f861c0b4d60 2024-03-25T19:40:23.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.764+0000 7f8620bdd640 1 -- 172.21.15.44:0/3525945241 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 411804047 0 0) 0x7f8608003520 con 0x7f861c0b1730 2024-03-25T19:40:23.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.764+0000 7f8620bdd640 1 -- 172.21.15.44:0/3525945241 >> v1:172.21.15.111:6789/0 conn(0x7f861c0b4d60 legacy=0x7f861c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:23.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.764+0000 7f8620bdd640 1 -- 172.21.15.44:0/3525945241 >> v1:172.21.15.44:6789/0 conn(0x7f861c14be90 legacy=0x7f861c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:23.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.764+0000 7f8620bdd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:23.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.764+0000 7f8620bdd640 1 monclient: found mon.noname-b 2024-03-25T19:40:23.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.764+0000 7f8620bdd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:23.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.764+0000 7f8620bdd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:40:23.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.764+0000 7f8620bdd640 1 -- 172.21.15.44:0/3525945241 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861c0b1030 con 0x7f861c0b1730 2024-03-25T19:40:23.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.764+0000 7f8620bdd640 10 monclient: _finish_auth 0 2024-03-25T19:40:23.766 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.764+0000 7f8620bdd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:23.766 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.764+0000 7f8620bdd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:23.766 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.764+0000 7f8620bdd640 1 -- 172.21.15.44:0/3525945241 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f860c001690 con 0x7f861c0b1730 2024-03-25T19:40:23.766 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.764+0000 7f8623e43640 5 monclient: authenticate success, global_id 4430 2024-03-25T19:40:23.766 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.764+0000 7f8623e43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:23.766 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.764+0000 7f8620bdd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:23.766 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.764+0000 7f8620bdd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:23.766 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.764+0000 7f8620bdd640 10 monclient: dump: 2024-03-25T19:40:23.766 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:23.766 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:23.766 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:23.766 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:23.766 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:23.767 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:23.767 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:23.767 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:23.767 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:23.767 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:23.767 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.764+0000 7f8623e43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:23.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.787+0000 7f8620bdd640 1 -- 172.21.15.44:0/3525945241 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f860c0017f0 con 0x7f861c0b1730 2024-03-25T19:40:23.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.787+0000 7f8620bdd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:23.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.787+0000 7f8620bdd640 1 -- 172.21.15.44:0/3525945241 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f860c001c00 con 0x7f861c0b1730 2024-03-25T19:40:23.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.787+0000 7f8623e43640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:23.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.787+0000 7f8623e43640 4 set_mon_vals no callback set 2024-03-25T19:40:23.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.787+0000 7f8620bdd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:23.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.787+0000 7f8620bdd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:23.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.787+0000 7f8620bdd640 10 monclient: dump: 2024-03-25T19:40:23.787 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:23.787 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:23.787 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:23.787 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:23.787 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:23.787 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:23.788 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:23.788 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:23.788 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:23.788 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:23.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.787+0000 7f8623e43640 10 monclient: shutdown 2024-03-25T19:40:23.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.787+0000 7f8623e43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:23.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.787+0000 7f8623e43640 1 -- 172.21.15.44:0/3525945241 >> v1:172.21.15.44:6790/0 conn(0x7f861c0b1730 legacy=0x7f861c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:23.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.787+0000 7f8623e43640 1 -- 172.21.15.44:0/3525945241 shutdown_connections 2024-03-25T19:40:23.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.787+0000 7f8623e43640 1 -- 172.21.15.44:0/3525945241 >> 172.21.15.44:0/3525945241 conn(0x7f861c0617a0 msgr2=0x7f861c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:23.789 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.788+0000 7f8623e43640 1 -- 172.21.15.44:0/3525945241 shutdown_connections 2024-03-25T19:40:23.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.788+0000 7f8623e43640 1 -- 172.21.15.44:0/3525945241 wait complete. 2024-03-25T19:40:23.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.788+0000 7f8623e43640 10 monclient: build_initial_monmap 2024-03-25T19:40:23.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.788+0000 7f8623e43640 10 monclient: monmap: 2024-03-25T19:40:23.790 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:40:23.790 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:23.790 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:40:23.790 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:40:23.790 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:23.790 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:40:23.790 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:40:23.790 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:40:23.790 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:40:23.790 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:23.790 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.789+0000 7f8623e43640 1 librados: starting msgr at 2024-03-25T19:40:23.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.789+0000 7f8623e43640 1 librados: starting objecter 2024-03-25T19:40:23.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.789+0000 7f8623e43640 1 Processor -- start 2024-03-25T19:40:23.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.789+0000 7f8623e43640 1 -- start start 2024-03-25T19:40:23.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.789+0000 7f8623e43640 1 librados: setting wanted keys 2024-03-25T19:40:23.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.789+0000 7f8623e43640 1 librados: calling monclient init 2024-03-25T19:40:23.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.789+0000 7f8623e43640 10 monclient: init 2024-03-25T19:40:23.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.790+0000 7f8623e43640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:23.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.790+0000 7f8623e43640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:23.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.790+0000 7f8623e43640 10 monclient(hunting): picked mon.noname-c con 0x7f861c0b1730 addr v1:172.21.15.44:6790/0 2024-03-25T19:40:23.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.790+0000 7f8623e43640 10 monclient(hunting): picked mon.noname-a con 0x7f861c0b4d60 addr v1:172.21.15.44:6789/0 2024-03-25T19:40:23.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.790+0000 7f8623e43640 10 monclient(hunting): picked mon.noname-b con 0x7f861c14be90 addr v1:172.21.15.111:6789/0 2024-03-25T19:40:23.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.790+0000 7f8623e43640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861c136250 con 0x7f861c0b4d60 2024-03-25T19:40:23.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.790+0000 7f8623e43640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861c136af0 con 0x7f861c14be90 2024-03-25T19:40:23.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.790+0000 7f8623e43640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861c136c80 con 0x7f861c0b1730 2024-03-25T19:40:23.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.790+0000 7f8623e43640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:23.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.790+0000 7f8623e43640 10 monclient(hunting): authenticate will time out at 5687.961168s 2024-03-25T19:40:23.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.795+0000 7f86223e0640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f861c14be90 0x7f861c1e2f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:46454/0 (socket says 172.21.15.44:46454) 2024-03-25T19:40:23.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.795+0000 7f86223e0640 1 -- 172.21.15.44:0/2768910808 learned_addr learned my addr 172.21.15.44:0/2768910808 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:40:23.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.796+0000 7f86127fc640 10 client.?.objecter ms_handle_connect 0x7f861c14be90 2024-03-25T19:40:23.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.796+0000 7f86127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:23.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.803+0000 7f86127fc640 1 -- 172.21.15.44:0/2768910808 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013310311 0 0) 0x7f861c136af0 con 0x7f861c14be90 2024-03-25T19:40:23.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.803+0000 7f86127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:23.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.803+0000 7f86127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:23.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.803+0000 7f86127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:23.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.803+0000 7f86127fc640 10 monclient(hunting): my global_id is 4444 2024-03-25T19:40:23.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.803+0000 7f86127fc640 1 -- 172.21.15.44:0/2768910808 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8600004eb0 con 0x7f861c14be90 2024-03-25T19:40:23.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.808+0000 7f86127fc640 10 client.?.objecter ms_handle_connect 0x7f861c0b1730 2024-03-25T19:40:23.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.808+0000 7f86127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:23.811 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.811+0000 7f86127fc640 10 client.?.objecter ms_handle_connect 0x7f861c0b4d60 2024-03-25T19:40:23.811 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.811+0000 7f86127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:23.815 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.815+0000 7f86127fc640 1 -- 172.21.15.44:0/2768910808 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690657745 0 0) 0x7f861c136c80 con 0x7f861c0b1730 2024-03-25T19:40:23.815 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.815+0000 7f86127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:23.815 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.815+0000 7f86127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:23.815 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.815+0000 7f86127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:23.815 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.815+0000 7f86127fc640 10 monclient(hunting): my global_id is 4433 2024-03-25T19:40:23.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.815+0000 7f86127fc640 1 -- 172.21.15.44:0/2768910808 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f861c136af0 con 0x7f861c0b1730 2024-03-25T19:40:23.817 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.817+0000 7f86127fc640 1 -- 172.21.15.44:0/2768910808 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3172118391 0 0) 0x7f8600004eb0 con 0x7f861c14be90 2024-03-25T19:40:23.817 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.817+0000 7f86127fc640 1 -- 172.21.15.44:0/2768910808 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f861c136c80 con 0x7f861c14be90 2024-03-25T19:40:23.817 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.817+0000 7f86127fc640 1 -- 172.21.15.44:0/2768910808 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f8614001c90 con 0x7f861c14be90 2024-03-25T19:40:23.817 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.817+0000 7f86127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:23.817 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.817+0000 7f86127fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:23.817 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.817+0000 7f86127fc640 10 monclient(hunting): dump: 2024-03-25T19:40:23.817 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:23.818 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:23.818 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:23.818 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:23.818 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:23.818 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:23.818 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:23.818 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:23.818 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:23.818 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:23.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.817+0000 7f86127fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:23.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.823+0000 7f86127fc640 1 -- 172.21.15.44:0/2768910808 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1685912493 0 0) 0x7f861c136c80 con 0x7f861c14be90 2024-03-25T19:40:23.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.824+0000 7f86127fc640 1 -- 172.21.15.44:0/2768910808 >> v1:172.21.15.44:6790/0 conn(0x7f861c0b1730 legacy=0x7f861c1d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:23.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.824+0000 7f86127fc640 1 -- 172.21.15.44:0/2768910808 >> v1:172.21.15.44:6789/0 conn(0x7f861c0b4d60 legacy=0x7f861c12fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:23.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.824+0000 7f86127fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:23.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.824+0000 7f86127fc640 1 monclient: found mon.b 2024-03-25T19:40:23.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.824+0000 7f86127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:23.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.824+0000 7f86127fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:40:23.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.824+0000 7f86127fc640 1 -- 172.21.15.44:0/2768910808 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861c138210 con 0x7f861c14be90 2024-03-25T19:40:23.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.824+0000 7f86127fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:23.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.824+0000 7f86127fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:23.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.824+0000 7f86127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:23.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.824+0000 7f8623e43640 5 monclient: authenticate success, global_id 4444 2024-03-25T19:40:23.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.824+0000 7f8623e43640 10 monclient: _renew_subs 2024-03-25T19:40:23.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.824+0000 7f8623e43640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:40:23.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.824+0000 7f8623e43640 1 -- 172.21.15.44:0/2768910808 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f861c1384c0 con 0x7f861c14be90 2024-03-25T19:40:23.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.824+0000 7f8623e43640 10 client.4444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:23.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.824+0000 7f8623e43640 10 monclient: _renew_subs 2024-03-25T19:40:23.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.824+0000 7f8623e43640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:40:23.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.824+0000 7f8623e43640 1 -- 172.21.15.44:0/2768910808 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f861c145700 con 0x7f861c14be90 2024-03-25T19:40:23.825 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.824+0000 7f8623e43640 1 librados: init done 2024-03-25T19:40:23.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.825+0000 7f8623e43640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:23.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.825+0000 7f8623e43640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:23.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.825+0000 7f8623e43640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:40:23.826 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:23.825+0000 7f8623e43640 1 -- 172.21.15.44:0/2768910808 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f861c1360a0 con 0x7f861c14be90 2024-03-25T19:40:24.093 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.093+0000 7f86127fc640 1 -- 172.21.15.44:0/2768910808 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8614003990 con 0x7f861c14be90 2024-03-25T19:40:24.093 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.093+0000 7f86127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:24.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.093+0000 7f86127fc640 1 -- 172.21.15.44:0/2768910808 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f8614003d60 con 0x7f861c14be90 2024-03-25T19:40:24.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.093+0000 7f86127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:24.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.093+0000 7f86127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:24.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.093+0000 7f86127fc640 10 monclient: dump: 2024-03-25T19:40:24.094 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:24.094 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:24.094 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:24.094 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:24.094 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:24.094 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:24.094 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:24.095 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:24.095 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:24.095 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:24.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.093+0000 7f8613fff640 4 set_mon_vals no callback set 2024-03-25T19:40:24.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.094+0000 7f86127fc640 1 -- 172.21.15.44:0/2768910808 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f8614003f80 con 0x7f861c14be90 2024-03-25T19:40:24.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.094+0000 7f8621bdf640 10 monclient: get_auth_request con 0x7f860006dd10 auth_method 0 2024-03-25T19:40:24.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.094+0000 7f86127fc640 10 client.4444.objecter ms_handle_connect 0x7f860006dd10 2024-03-25T19:40:24.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.099+0000 7f86127fc640 1 -- 172.21.15.44:0/2768910808 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f86140549d0 con 0x7f861c14be90 2024-03-25T19:40:24.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.099+0000 7f86127fc640 10 client.4444.objecter ms_dispatch 0x7f861c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:24.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.099+0000 7f86127fc640 3 client.4444.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:24.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.099+0000 7f86127fc640 3 client.4444.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:24.100 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.099+0000 7f86127fc640 20 client.4444.objecter dump_active .. 0 homeless 2024-03-25T19:40:24.100 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.100+0000 7f86127fc640 1 -- 172.21.15.44:0/2768910808 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f86140504e0 con 0x7f861c14be90 2024-03-25T19:40:24.100 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.100+0000 7f86127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:24.100 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.100+0000 7f86127fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:24.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.338+0000 7f8623e43640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:24.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.338+0000 7f8623e43640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:24.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.338+0000 7f8623e43640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:40:24.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.338+0000 7f8623e43640 1 -- 172.21.15.44:0/2768910808 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f861c135ec0 con 0x7f861c14be90 2024-03-25T19:40:24.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.413+0000 7f86127fc640 1 -- 172.21.15.44:0/2768910808 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f8614052410 con 0x7f861c14be90 2024-03-25T19:40:24.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.413+0000 7f86127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:24.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.413+0000 7f86127fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:24.413 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:40:24.414 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:24.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.416+0000 7f8623e43640 10 librados: watch_flush enter 2024-03-25T19:40:24.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.416+0000 7f8623e43640 10 librados: watch_flush exit 2024-03-25T19:40:24.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.416+0000 7f8623e43640 20 client.4444.objecter shutdown clearing up homeless session... 2024-03-25T19:40:24.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.416+0000 7f8623e43640 10 client.4444.objecter successfully canceled tick 2024-03-25T19:40:24.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.416+0000 7f8623e43640 1 -- 172.21.15.44:0/2768910808 >> v1:172.21.15.111:6800/3883215793 conn(0x7f860006dd10 legacy=0x7f86000701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:24.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.416+0000 7f8623e43640 10 monclient: shutdown 2024-03-25T19:40:24.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.416+0000 7f8623e43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:24.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.416+0000 7f8623e43640 1 -- 172.21.15.44:0/2768910808 >> v1:172.21.15.111:6789/0 conn(0x7f861c14be90 legacy=0x7f861c1e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:24.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.416+0000 7f8623e43640 1 -- 172.21.15.44:0/2768910808 shutdown_connections 2024-03-25T19:40:24.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.416+0000 7f8623e43640 1 -- 172.21.15.44:0/2768910808 >> 172.21.15.44:0/2768910808 conn(0x7f861c0617a0 msgr2=0x7f861c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:24.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.416+0000 7f8623e43640 1 -- 172.21.15.44:0/2768910808 shutdown_connections 2024-03-25T19:40:24.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.416+0000 7f8623e43640 1 -- 172.21.15.44:0/2768910808 wait complete. 2024-03-25T19:40:24.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:24.416+0000 7f8623e43640 1 librados: shutdown 2024-03-25T19:40:24.428 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:31.234 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:31.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.353+0000 7f23df7af640 10 monclient: get_monmap_and_config 2024-03-25T19:40:31.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.353+0000 7f23df7af640 10 monclient: build_initial_monmap 2024-03-25T19:40:31.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.353+0000 7f23df7af640 10 monclient: monmap: 2024-03-25T19:40:31.354 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:40:31.354 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:31.354 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:40:31.354627+0000 2024-03-25T19:40:31.354 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:40:31.354627+0000 2024-03-25T19:40:31.354 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:31.354 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:31.354 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:40:31.354 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:40:31.354 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:40:31.354 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:31.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.354+0000 7f23df7af640 1 Processor -- start 2024-03-25T19:40:31.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.354+0000 7f23df7af640 1 -- start start 2024-03-25T19:40:31.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.354+0000 7f23df7af640 10 monclient: init 2024-03-25T19:40:31.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.355+0000 7f23df7af640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:31.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.355+0000 7f23df7af640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:31.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.355+0000 7f23df7af640 10 monclient(hunting): picked mon.noname-b con 0x7f23d80a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T19:40:31.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.355+0000 7f23df7af640 10 monclient(hunting): picked mon.noname-a con 0x7f23d813f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:40:31.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.355+0000 7f23df7af640 10 monclient(hunting): picked mon.noname-c con 0x7f23d81430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:40:31.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.355+0000 7f23df7af640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d8142520 con 0x7f23d813f710 2024-03-25T19:40:31.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.355+0000 7f23df7af640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d80b28a0 con 0x7f23d81430c0 2024-03-25T19:40:31.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.355+0000 7f23df7af640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d80b2a30 con 0x7f23d80a98f0 2024-03-25T19:40:31.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.355+0000 7f23df7af640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:31.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.355+0000 7f23df7af640 10 monclient(hunting): authenticate will time out at 5695.526324s 2024-03-25T19:40:31.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.355+0000 7f23ddd4c640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f23d81430c0 0x7f23d814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:56314/0 (socket says 172.21.15.44:56314) 2024-03-25T19:40:31.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.355+0000 7f23ddd4c640 1 -- 172.21.15.44:0/1189857729 learned_addr learned my addr 172.21.15.44:0/1189857729 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:40:31.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.441+0000 7f23cffff640 1 -- 172.21.15.44:0/1189857729 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2894256012 0 0) 0x7f23d80b2a30 con 0x7f23d80a98f0 2024-03-25T19:40:31.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.441+0000 7f23cffff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:31.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.441+0000 7f23cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:31.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.441+0000 7f23cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:31.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.441+0000 7f23cffff640 10 monclient(hunting): my global_id is 4436 2024-03-25T19:40:31.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.441+0000 7f23cffff640 1 -- 172.21.15.44:0/1189857729 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c4003520 con 0x7f23d80a98f0 2024-03-25T19:40:31.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.447+0000 7f23cffff640 1 -- 172.21.15.44:0/1189857729 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079845898 0 0) 0x7f23d80b28a0 con 0x7f23d81430c0 2024-03-25T19:40:31.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.447+0000 7f23cffff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:31.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.447+0000 7f23cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:31.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.447+0000 7f23cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:31.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.447+0000 7f23cffff640 10 monclient(hunting): my global_id is 4447 2024-03-25T19:40:31.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.447+0000 7f23cffff640 1 -- 172.21.15.44:0/1189857729 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23d80b2a30 con 0x7f23d81430c0 2024-03-25T19:40:31.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.461+0000 7f23cffff640 1 -- 172.21.15.44:0/1189857729 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2312189443 0 0) 0x7f23c4003520 con 0x7f23d80a98f0 2024-03-25T19:40:31.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.461+0000 7f23cffff640 1 -- 172.21.15.44:0/1189857729 >> v1:172.21.15.111:6789/0 conn(0x7f23d81430c0 legacy=0x7f23d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:31.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.461+0000 7f23cffff640 1 -- 172.21.15.44:0/1189857729 >> v1:172.21.15.44:6789/0 conn(0x7f23d813f710 legacy=0x7f23d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:31.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.461+0000 7f23cffff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:31.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.461+0000 7f23cffff640 1 monclient: found mon.noname-b 2024-03-25T19:40:31.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.461+0000 7f23cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:31.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.462+0000 7f23cffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:40:31.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.462+0000 7f23cffff640 1 -- 172.21.15.44:0/1189857729 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d80b44c0 con 0x7f23d80a98f0 2024-03-25T19:40:31.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.462+0000 7f23cffff640 10 monclient: _finish_auth 0 2024-03-25T19:40:31.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.462+0000 7f23cffff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:31.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.462+0000 7f23cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:31.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.462+0000 7f23cffff640 1 -- 172.21.15.44:0/1189857729 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f23d4001690 con 0x7f23d80a98f0 2024-03-25T19:40:31.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.462+0000 7f23df7af640 5 monclient: authenticate success, global_id 4436 2024-03-25T19:40:31.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.462+0000 7f23df7af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:31.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.462+0000 7f23cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:31.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.462+0000 7f23cffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:31.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.462+0000 7f23cffff640 10 monclient: dump: 2024-03-25T19:40:31.463 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:31.463 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:31.463 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:31.463 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:31.463 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:31.463 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:31.463 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:31.464 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:31.464 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:31.464 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:31.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.462+0000 7f23df7af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:31.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.484+0000 7f23cffff640 1 -- 172.21.15.44:0/1189857729 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23d40017f0 con 0x7f23d80a98f0 2024-03-25T19:40:31.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.484+0000 7f23cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:31.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.484+0000 7f23cffff640 1 -- 172.21.15.44:0/1189857729 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f23d4001c00 con 0x7f23d80a98f0 2024-03-25T19:40:31.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.484+0000 7f23cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:31.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.484+0000 7f23cffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:31.485 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.484+0000 7f23cffff640 10 monclient: dump: 2024-03-25T19:40:31.485 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:31.485 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:31.485 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:31.485 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:31.485 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:31.485 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:31.485 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:31.485 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:31.485 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:31.486 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:31.486 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.484+0000 7f23df7af640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:31.486 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.484+0000 7f23df7af640 4 set_mon_vals no callback set 2024-03-25T19:40:31.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.485+0000 7f23df7af640 10 monclient: shutdown 2024-03-25T19:40:31.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.485+0000 7f23df7af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:31.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.485+0000 7f23df7af640 1 -- 172.21.15.44:0/1189857729 >> v1:172.21.15.44:6790/0 conn(0x7f23d80a98f0 legacy=0x7f23d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:31.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.485+0000 7f23df7af640 1 -- 172.21.15.44:0/1189857729 shutdown_connections 2024-03-25T19:40:31.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.485+0000 7f23df7af640 1 -- 172.21.15.44:0/1189857729 >> 172.21.15.44:0/1189857729 conn(0x7f23d80b7ea0 msgr2=0x7f23d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:31.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.485+0000 7f23df7af640 1 -- 172.21.15.44:0/1189857729 shutdown_connections 2024-03-25T19:40:31.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.485+0000 7f23df7af640 1 -- 172.21.15.44:0/1189857729 wait complete. 2024-03-25T19:40:31.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.486+0000 7f23df7af640 10 monclient: build_initial_monmap 2024-03-25T19:40:31.489 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.486+0000 7f23df7af640 10 monclient: monmap: 2024-03-25T19:40:31.489 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:40:31.489 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:31.489 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:40:31.489 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:40:31.489 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:31.489 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:40:31.489 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:40:31.489 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:40:31.490 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:40:31.490 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:31.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.486+0000 7f23df7af640 1 librados: starting msgr at 2024-03-25T19:40:31.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.486+0000 7f23df7af640 1 librados: starting objecter 2024-03-25T19:40:31.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.487+0000 7f23df7af640 1 Processor -- start 2024-03-25T19:40:31.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.487+0000 7f23df7af640 1 -- start start 2024-03-25T19:40:31.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.487+0000 7f23df7af640 1 librados: setting wanted keys 2024-03-25T19:40:31.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.487+0000 7f23df7af640 1 librados: calling monclient init 2024-03-25T19:40:31.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.487+0000 7f23df7af640 10 monclient: init 2024-03-25T19:40:31.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.487+0000 7f23df7af640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:31.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.487+0000 7f23df7af640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:31.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.487+0000 7f23df7af640 10 monclient(hunting): picked mon.noname-a con 0x7f23d80a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:40:31.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.487+0000 7f23df7af640 10 monclient(hunting): picked mon.noname-b con 0x7f23d813f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:40:31.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.487+0000 7f23df7af640 10 monclient(hunting): picked mon.noname-c con 0x7f23d81430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:40:31.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.487+0000 7f23df7af640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d8137370 con 0x7f23d80a98f0 2024-03-25T19:40:31.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.487+0000 7f23df7af640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d81352a0 con 0x7f23d813f710 2024-03-25T19:40:31.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.487+0000 7f23df7af640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d8131760 con 0x7f23d81430c0 2024-03-25T19:40:31.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.487+0000 7f23df7af640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:31.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.487+0000 7f23df7af640 10 monclient(hunting): authenticate will time out at 5695.658813s 2024-03-25T19:40:31.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.495+0000 7f23dcd4a640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f23d813f710 0x7f23d81d7750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:56356/0 (socket says 172.21.15.44:56356) 2024-03-25T19:40:31.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.495+0000 7f23dcd4a640 1 -- 172.21.15.44:0/329990076 learned_addr learned my addr 172.21.15.44:0/329990076 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:40:31.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.496+0000 7f23cdffb640 10 client.?.objecter ms_handle_connect 0x7f23d813f710 2024-03-25T19:40:31.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.496+0000 7f23cdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:31.498 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.498+0000 7f23cdffb640 10 client.?.objecter ms_handle_connect 0x7f23d81430c0 2024-03-25T19:40:31.498 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.498+0000 7f23cdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:31.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.500+0000 7f23cdffb640 1 -- 172.21.15.44:0/329990076 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314599056 0 0) 0x7f23d81352a0 con 0x7f23d813f710 2024-03-25T19:40:31.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.500+0000 7f23cdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:31.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.500+0000 7f23cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:31.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.500+0000 7f23cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:31.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.500+0000 7f23cdffb640 10 monclient(hunting): my global_id is 4450 2024-03-25T19:40:31.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.500+0000 7f23cdffb640 1 -- 172.21.15.44:0/329990076 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23bc004eb0 con 0x7f23d813f710 2024-03-25T19:40:31.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.506+0000 7f23cdffb640 1 -- 172.21.15.44:0/329990076 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757865774 0 0) 0x7f23d8131760 con 0x7f23d81430c0 2024-03-25T19:40:31.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.506+0000 7f23cdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:31.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.506+0000 7f23cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:31.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.506+0000 7f23cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:31.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.506+0000 7f23cdffb640 10 monclient(hunting): my global_id is 4439 2024-03-25T19:40:31.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.506+0000 7f23cdffb640 1 -- 172.21.15.44:0/329990076 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23d81352a0 con 0x7f23d81430c0 2024-03-25T19:40:31.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.507+0000 7f23cdffb640 10 client.?.objecter ms_handle_connect 0x7f23d80a98f0 2024-03-25T19:40:31.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.507+0000 7f23cdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:31.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.513+0000 7f23cdffb640 1 -- 172.21.15.44:0/329990076 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3194952775 0 0) 0x7f23bc004eb0 con 0x7f23d813f710 2024-03-25T19:40:31.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.513+0000 7f23cdffb640 1 -- 172.21.15.44:0/329990076 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23d8131760 con 0x7f23d813f710 2024-03-25T19:40:31.514 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.513+0000 7f23cdffb640 1 -- 172.21.15.44:0/329990076 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f23c80019d0 con 0x7f23d813f710 2024-03-25T19:40:31.514 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.513+0000 7f23cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:31.514 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.513+0000 7f23cdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:31.514 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.513+0000 7f23cdffb640 10 monclient(hunting): dump: 2024-03-25T19:40:31.514 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:31.514 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:31.514 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:31.514 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:31.514 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:31.514 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:31.514 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:31.514 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:31.514 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:31.514 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:31.515 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.513+0000 7f23cdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:31.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.525+0000 7f23cdffb640 1 -- 172.21.15.44:0/329990076 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2910532422 0 0) 0x7f23d8131760 con 0x7f23d813f710 2024-03-25T19:40:31.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.525+0000 7f23cdffb640 1 -- 172.21.15.44:0/329990076 >> v1:172.21.15.44:6790/0 conn(0x7f23d81430c0 legacy=0x7f23d81d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:31.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.525+0000 7f23cdffb640 1 -- 172.21.15.44:0/329990076 >> v1:172.21.15.44:6789/0 conn(0x7f23d80a98f0 legacy=0x7f23d8131050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:31.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.525+0000 7f23cdffb640 10 monclient: _finish_hunting 0 2024-03-25T19:40:31.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.525+0000 7f23cdffb640 1 monclient: found mon.b 2024-03-25T19:40:31.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.525+0000 7f23cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:31.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.525+0000 7f23cdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:40:31.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.525+0000 7f23cdffb640 1 -- 172.21.15.44:0/329990076 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d81318f0 con 0x7f23d813f710 2024-03-25T19:40:31.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.525+0000 7f23cdffb640 10 monclient: _finish_auth 0 2024-03-25T19:40:31.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.525+0000 7f23cdffb640 10 monclient: _check_auth_tickets 2024-03-25T19:40:31.526 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.525+0000 7f23cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:31.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.525+0000 7f23df7af640 5 monclient: authenticate success, global_id 4450 2024-03-25T19:40:31.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.525+0000 7f23df7af640 10 monclient: _renew_subs 2024-03-25T19:40:31.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.525+0000 7f23df7af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:40:31.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.525+0000 7f23df7af640 1 -- 172.21.15.44:0/329990076 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d80b7a40 con 0x7f23d813f710 2024-03-25T19:40:31.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.526+0000 7f23df7af640 10 client.4450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:31.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.526+0000 7f23df7af640 10 monclient: _renew_subs 2024-03-25T19:40:31.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.526+0000 7f23df7af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:40:31.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.526+0000 7f23df7af640 1 -- 172.21.15.44:0/329990076 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d81360e0 con 0x7f23d813f710 2024-03-25T19:40:31.527 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.526+0000 7f23df7af640 1 librados: init done 2024-03-25T19:40:31.528 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.527+0000 7f23df7af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:31.528 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.527+0000 7f23df7af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:31.528 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.527+0000 7f23df7af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:40:31.528 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.527+0000 7f23df7af640 1 -- 172.21.15.44:0/329990076 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d8135430 con 0x7f23d813f710 2024-03-25T19:40:31.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.540+0000 7f23cdffb640 1 -- 172.21.15.44:0/329990076 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23c80035f0 con 0x7f23d813f710 2024-03-25T19:40:31.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.540+0000 7f23cdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:31.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.540+0000 7f23cdffb640 1 -- 172.21.15.44:0/329990076 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f23c8003a00 con 0x7f23d813f710 2024-03-25T19:40:31.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.541+0000 7f23cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:31.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.541+0000 7f23cdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:31.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.541+0000 7f23cdffb640 10 monclient: dump: 2024-03-25T19:40:31.541 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:31.541 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:31.541 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:31.541 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:31.541 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:31.541 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:31.541 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:31.541 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:31.542 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:31.542 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:31.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.541+0000 7f23cf7fe640 4 set_mon_vals no callback set 2024-03-25T19:40:31.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.542+0000 7f23cdffb640 1 -- 172.21.15.44:0/329990076 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f23c8003c20 con 0x7f23d813f710 2024-03-25T19:40:31.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.543+0000 7f23dd54b640 10 monclient: get_auth_request con 0x7f23bc06dd10 auth_method 0 2024-03-25T19:40:31.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.543+0000 7f23cdffb640 10 client.4450.objecter ms_handle_connect 0x7f23bc06dd10 2024-03-25T19:40:31.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.545+0000 7f23cdffb640 1 -- 172.21.15.44:0/329990076 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f23c80547d0 con 0x7f23d813f710 2024-03-25T19:40:31.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.545+0000 7f23cdffb640 10 client.4450.objecter ms_dispatch 0x7f23d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:31.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.545+0000 7f23cdffb640 3 client.4450.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:31.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.545+0000 7f23cdffb640 3 client.4450.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:31.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.545+0000 7f23cdffb640 20 client.4450.objecter dump_active .. 0 homeless 2024-03-25T19:40:31.811 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.811+0000 7f23cdffb640 1 -- 172.21.15.44:0/329990076 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23c80502e0 con 0x7f23d813f710 2024-03-25T19:40:31.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.811+0000 7f23cdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:31.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:31.811+0000 7f23cdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:32.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:32.065+0000 7f23df7af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:32.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:32.066+0000 7f23df7af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:32.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:32.066+0000 7f23df7af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:40:32.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:32.066+0000 7f23df7af640 1 -- 172.21.15.44:0/329990076 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23d80627b0 con 0x7f23d813f710 2024-03-25T19:40:32.081 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:32.082+0000 7f23cdffb640 1 -- 172.21.15.44:0/329990076 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f23c8052210 con 0x7f23d813f710 2024-03-25T19:40:32.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:32.082+0000 7f23cdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:32.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:32.082+0000 7f23cdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:32.082 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:40:32.082 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:32.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:32.084+0000 7f23df7af640 10 librados: watch_flush enter 2024-03-25T19:40:32.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:32.084+0000 7f23df7af640 10 librados: watch_flush exit 2024-03-25T19:40:32.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:32.084+0000 7f23df7af640 20 client.4450.objecter shutdown clearing up homeless session... 2024-03-25T19:40:32.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:32.084+0000 7f23df7af640 10 client.4450.objecter successfully canceled tick 2024-03-25T19:40:32.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:32.084+0000 7f23df7af640 1 -- 172.21.15.44:0/329990076 >> v1:172.21.15.111:6800/3883215793 conn(0x7f23bc06dd10 legacy=0x7f23bc0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:32.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:32.084+0000 7f23df7af640 10 monclient: shutdown 2024-03-25T19:40:32.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:32.084+0000 7f23df7af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:32.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:32.084+0000 7f23df7af640 1 -- 172.21.15.44:0/329990076 >> v1:172.21.15.111:6789/0 conn(0x7f23d813f710 legacy=0x7f23d81d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:32.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:32.085+0000 7f23df7af640 1 -- 172.21.15.44:0/329990076 shutdown_connections 2024-03-25T19:40:32.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:32.085+0000 7f23df7af640 1 -- 172.21.15.44:0/329990076 >> 172.21.15.44:0/329990076 conn(0x7f23d80b7ea0 msgr2=0x7f23d8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:32.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:32.085+0000 7f23df7af640 1 -- 172.21.15.44:0/329990076 shutdown_connections 2024-03-25T19:40:32.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:32.085+0000 7f23df7af640 1 -- 172.21.15.44:0/329990076 wait complete. 2024-03-25T19:40:32.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:32.085+0000 7f23df7af640 1 librados: shutdown 2024-03-25T19:40:32.097 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:38.903 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:39.023 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.024+0000 7f8e4ad3b640 10 monclient: get_monmap_and_config 2024-03-25T19:40:39.023 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.024+0000 7f8e4ad3b640 10 monclient: build_initial_monmap 2024-03-25T19:40:39.023 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.024+0000 7f8e4ad3b640 10 monclient: monmap: 2024-03-25T19:40:39.023 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:40:39.023 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:39.024 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:40:39.024723+0000 2024-03-25T19:40:39.024 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:40:39.024723+0000 2024-03-25T19:40:39.024 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:39.024 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:39.024 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:40:39.024 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:40:39.024 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:40:39.024 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:39.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.025+0000 7f8e4ad3b640 1 Processor -- start 2024-03-25T19:40:39.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.025+0000 7f8e4ad3b640 1 -- start start 2024-03-25T19:40:39.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.025+0000 7f8e4ad3b640 10 monclient: init 2024-03-25T19:40:39.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.025+0000 7f8e4ad3b640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:39.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.025+0000 7f8e4ad3b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:39.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.025+0000 7f8e4ad3b640 10 monclient(hunting): picked mon.noname-a con 0x7f8e440a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:40:39.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.025+0000 7f8e4ad3b640 10 monclient(hunting): picked mon.noname-c con 0x7f8e4413f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:40:39.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.025+0000 7f8e4ad3b640 10 monclient(hunting): picked mon.noname-b con 0x7f8e441430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:40:39.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.026+0000 7f8e4ad3b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e44142520 con 0x7f8e440a98f0 2024-03-25T19:40:39.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.026+0000 7f8e4ad3b640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e440b28a0 con 0x7f8e4413f710 2024-03-25T19:40:39.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.026+0000 7f8e4ad3b640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e440b2a30 con 0x7f8e441430c0 2024-03-25T19:40:39.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.026+0000 7f8e4ad3b640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:39.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.026+0000 7f8e4ad3b640 10 monclient(hunting): authenticate will time out at 5703.195878s 2024-03-25T19:40:39.037 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.038+0000 7f8e492d8640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f8e441430c0 0x7f8e4414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:55110/0 (socket says 172.21.15.44:55110) 2024-03-25T19:40:39.037 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.038+0000 7f8e492d8640 1 -- 172.21.15.44:0/3993071562 learned_addr learned my addr 172.21.15.44:0/3993071562 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:40:39.048 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.049+0000 7f8e437fe640 1 -- 172.21.15.44:0/3993071562 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1140069019 0 0) 0x7f8e440b2a30 con 0x7f8e441430c0 2024-03-25T19:40:39.048 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.049+0000 7f8e437fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:39.048 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.049+0000 7f8e437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:39.048 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.049+0000 7f8e437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:39.048 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.049+0000 7f8e437fe640 10 monclient(hunting): my global_id is 4442 2024-03-25T19:40:39.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.049+0000 7f8e437fe640 1 -- 172.21.15.44:0/3993071562 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e34003520 con 0x7f8e441430c0 2024-03-25T19:40:39.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.052+0000 7f8e48ad7640 1 -- 172.21.15.44:0/3993071562 >> v1:172.21.15.44:6789/0 conn(0x7f8e440a98f0 legacy=0x7f8e440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-25T19:40:39.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.052+0000 7f8e48ad7640 1 -- 172.21.15.44:0/3993071562 >> v1:172.21.15.44:6789/0 conn(0x7f8e440a98f0 legacy=0x7f8e440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:40:39.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.052+0000 7f8e48ad7640 1 --1- 172.21.15.44:0/3993071562 >> v1:172.21.15.44:6789/0 conn(0x7f8e440a98f0 0x7f8e440a9cd0 :-1 s=OPENED pgs=35735 cs=1 l=1).handle_message read tag failed 2024-03-25T19:40:39.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.052+0000 7f8e48ad7640 1 --1- 172.21.15.44:0/3993071562 >> v1:172.21.15.44:6789/0 conn(0x7f8e440a98f0 0x7f8e440a9cd0 :-1 s=OPENED pgs=35735 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:40:39.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.052+0000 7f8e437fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.44:6789/0 2024-03-25T19:40:39.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.058+0000 7f8e437fe640 1 -- 172.21.15.44:0/3993071562 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3301175748 0 0) 0x7f8e440b28a0 con 0x7f8e4413f710 2024-03-25T19:40:39.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.058+0000 7f8e437fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:39.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.058+0000 7f8e437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:39.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.058+0000 7f8e437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:39.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.058+0000 7f8e437fe640 10 monclient(hunting): my global_id is 4453 2024-03-25T19:40:39.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.058+0000 7f8e437fe640 1 -- 172.21.15.44:0/3993071562 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e440b2a30 con 0x7f8e4413f710 2024-03-25T19:40:39.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.066+0000 7f8e437fe640 1 -- 172.21.15.44:0/3993071562 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 935774131 0 0) 0x7f8e440b2a30 con 0x7f8e4413f710 2024-03-25T19:40:39.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.066+0000 7f8e437fe640 1 -- 172.21.15.44:0/3993071562 >> v1:172.21.15.44:6790/0 conn(0x7f8e441430c0 legacy=0x7f8e4414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:39.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.066+0000 7f8e437fe640 1 -- 172.21.15.44:0/3993071562 >> v1:172.21.15.44:6789/0 conn(0x7f8e440a98f0 legacy=0x7f8e440a9cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:40:39.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.066+0000 7f8e437fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:39.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.066+0000 7f8e437fe640 1 monclient: found mon.noname-c 2024-03-25T19:40:39.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.066+0000 7f8e437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:39.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.066+0000 7f8e437fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:40:39.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.066+0000 7f8e437fe640 1 -- 172.21.15.44:0/3993071562 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e440b44c0 con 0x7f8e4413f710 2024-03-25T19:40:39.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.066+0000 7f8e437fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:39.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.066+0000 7f8e437fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:39.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.066+0000 7f8e437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:39.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.066+0000 7f8e437fe640 1 -- 172.21.15.44:0/3993071562 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f8e30001630 con 0x7f8e4413f710 2024-03-25T19:40:39.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.066+0000 7f8e437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:39.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.066+0000 7f8e437fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:39.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.066+0000 7f8e437fe640 10 monclient: dump: 2024-03-25T19:40:39.066 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:39.066 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:39.066 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:39.066 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:39.066 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:39.067 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:39.067 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:39.067 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:39.067 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:39.067 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:39.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.066+0000 7f8e4ad3b640 5 monclient: authenticate success, global_id 4453 2024-03-25T19:40:39.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.066+0000 7f8e4ad3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:39.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.071+0000 7f8e437fe640 1 -- 172.21.15.44:0/3993071562 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e30001790 con 0x7f8e4413f710 2024-03-25T19:40:39.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.071+0000 7f8e437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:39.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.071+0000 7f8e437fe640 1 -- 172.21.15.44:0/3993071562 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f8e30001ba0 con 0x7f8e4413f710 2024-03-25T19:40:39.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.071+0000 7f8e437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:39.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.071+0000 7f8e437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:39.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.071+0000 7f8e437fe640 10 monclient: dump: 2024-03-25T19:40:39.070 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:39.071 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:39.071 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:39.071 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:39.071 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:39.071 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:39.071 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:39.071 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:39.071 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:39.071 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:39.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.071+0000 7f8e4ad3b640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:39.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.071+0000 7f8e4ad3b640 4 set_mon_vals no callback set 2024-03-25T19:40:39.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.072+0000 7f8e4ad3b640 10 monclient: shutdown 2024-03-25T19:40:39.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.072+0000 7f8e4ad3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:39.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.072+0000 7f8e4ad3b640 1 -- 172.21.15.44:0/3993071562 >> v1:172.21.15.111:6789/0 conn(0x7f8e4413f710 legacy=0x7f8e441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:39.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.072+0000 7f8e4ad3b640 1 -- 172.21.15.44:0/3993071562 shutdown_connections 2024-03-25T19:40:39.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.072+0000 7f8e4ad3b640 1 -- 172.21.15.44:0/3993071562 >> 172.21.15.44:0/3993071562 conn(0x7f8e440b7ea0 msgr2=0x7f8e440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:39.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.072+0000 7f8e4ad3b640 1 -- 172.21.15.44:0/3993071562 shutdown_connections 2024-03-25T19:40:39.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.072+0000 7f8e4ad3b640 1 -- 172.21.15.44:0/3993071562 wait complete. 2024-03-25T19:40:39.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.073+0000 7f8e4ad3b640 10 monclient: build_initial_monmap 2024-03-25T19:40:39.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.073+0000 7f8e4ad3b640 10 monclient: monmap: 2024-03-25T19:40:39.073 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:40:39.073 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:39.073 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:40:39.073 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:40:39.073 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:39.074 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:40:39.074 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:40:39.074 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:40:39.074 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:40:39.074 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:39.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.073+0000 7f8e4ad3b640 1 librados: starting msgr at 2024-03-25T19:40:39.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.073+0000 7f8e4ad3b640 1 librados: starting objecter 2024-03-25T19:40:39.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.073+0000 7f8e4ad3b640 1 Processor -- start 2024-03-25T19:40:39.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.073+0000 7f8e4ad3b640 1 -- start start 2024-03-25T19:40:39.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.073+0000 7f8e4ad3b640 1 librados: setting wanted keys 2024-03-25T19:40:39.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.073+0000 7f8e4ad3b640 1 librados: calling monclient init 2024-03-25T19:40:39.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.073+0000 7f8e4ad3b640 10 monclient: init 2024-03-25T19:40:39.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.074+0000 7f8e4ad3b640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:39.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.074+0000 7f8e4ad3b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:39.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.074+0000 7f8e4ad3b640 10 monclient(hunting): picked mon.noname-b con 0x7f8e440a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:40:39.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.074+0000 7f8e4ad3b640 10 monclient(hunting): picked mon.noname-c con 0x7f8e4413f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:40:39.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.074+0000 7f8e4ad3b640 10 monclient(hunting): picked mon.noname-a con 0x7f8e441430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:40:39.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.074+0000 7f8e4ad3b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e440abcc0 con 0x7f8e441430c0 2024-03-25T19:40:39.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.075+0000 7f8e4ad3b640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e441d4540 con 0x7f8e440a98f0 2024-03-25T19:40:39.076 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.075+0000 7f8e4ad3b640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e441d83d0 con 0x7f8e4413f710 2024-03-25T19:40:39.076 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.075+0000 7f8e4ad3b640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:39.076 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.075+0000 7f8e4ad3b640 10 monclient(hunting): authenticate will time out at 5703.244925s 2024-03-25T19:40:39.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.087+0000 7f8e43fff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f8e4413f710 0x7f8e441d4990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:55164/0 (socket says 172.21.15.44:55164) 2024-03-25T19:40:39.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.087+0000 7f8e43fff640 1 -- 172.21.15.44:0/1599373760 learned_addr learned my addr 172.21.15.44:0/1599373760 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:40:39.090 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.091+0000 7f8e417fa640 10 client.?.objecter ms_handle_connect 0x7f8e440a98f0 2024-03-25T19:40:39.090 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.091+0000 7f8e417fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:39.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.092+0000 7f8e417fa640 10 client.?.objecter ms_handle_connect 0x7f8e4413f710 2024-03-25T19:40:39.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.092+0000 7f8e417fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:39.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.096+0000 7f8e417fa640 10 client.?.objecter ms_handle_connect 0x7f8e441430c0 2024-03-25T19:40:39.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.096+0000 7f8e417fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:39.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.103+0000 7f8e417fa640 1 -- 172.21.15.44:0/1599373760 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30248711 0 0) 0x7f8e441d4540 con 0x7f8e440a98f0 2024-03-25T19:40:39.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.103+0000 7f8e417fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:39.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.103+0000 7f8e417fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:39.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.103+0000 7f8e417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:39.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.103+0000 7f8e417fa640 10 monclient(hunting): my global_id is 4456 2024-03-25T19:40:39.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.103+0000 7f8e417fa640 1 -- 172.21.15.44:0/1599373760 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e28004eb0 con 0x7f8e440a98f0 2024-03-25T19:40:39.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.110+0000 7f8e417fa640 1 -- 172.21.15.44:0/1599373760 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146123396 0 0) 0x7f8e441d83d0 con 0x7f8e4413f710 2024-03-25T19:40:39.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.110+0000 7f8e417fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:39.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.110+0000 7f8e417fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:39.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.110+0000 7f8e417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:39.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.110+0000 7f8e417fa640 10 monclient(hunting): my global_id is 4445 2024-03-25T19:40:39.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.111+0000 7f8e417fa640 1 -- 172.21.15.44:0/1599373760 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e441d4540 con 0x7f8e4413f710 2024-03-25T19:40:39.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.112+0000 7f8e417fa640 1 -- 172.21.15.44:0/1599373760 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3136675095 0 0) 0x7f8e440abcc0 con 0x7f8e441430c0 2024-03-25T19:40:39.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.112+0000 7f8e417fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:39.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.112+0000 7f8e417fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:39.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.112+0000 7f8e417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:39.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.112+0000 7f8e417fa640 10 monclient(hunting): my global_id is 4473 2024-03-25T19:40:39.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.112+0000 7f8e417fa640 1 -- 172.21.15.44:0/1599373760 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e441d83d0 con 0x7f8e441430c0 2024-03-25T19:40:39.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.114+0000 7f8e417fa640 1 -- 172.21.15.44:0/1599373760 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2230339156 0 0) 0x7f8e28004eb0 con 0x7f8e440a98f0 2024-03-25T19:40:39.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.115+0000 7f8e417fa640 1 -- 172.21.15.44:0/1599373760 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e440abcc0 con 0x7f8e440a98f0 2024-03-25T19:40:39.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.115+0000 7f8e417fa640 1 -- 172.21.15.44:0/1599373760 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f8e38001960 con 0x7f8e440a98f0 2024-03-25T19:40:39.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.115+0000 7f8e417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:39.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.115+0000 7f8e417fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:39.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.115+0000 7f8e417fa640 10 monclient(hunting): dump: 2024-03-25T19:40:39.114 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:39.114 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:39.114 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:39.114 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:39.114 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:39.114 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:39.115 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:39.115 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:39.115 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:39.115 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:39.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.115+0000 7f8e417fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:39.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.127+0000 7f8e417fa640 1 -- 172.21.15.44:0/1599373760 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1378129793 0 0) 0x7f8e441d83d0 con 0x7f8e441430c0 2024-03-25T19:40:39.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.127+0000 7f8e417fa640 1 -- 172.21.15.44:0/1599373760 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e28004eb0 con 0x7f8e441430c0 2024-03-25T19:40:39.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.127+0000 7f8e417fa640 1 -- 172.21.15.44:0/1599373760 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f8e3c001a30 con 0x7f8e441430c0 2024-03-25T19:40:39.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.127+0000 7f8e417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:39.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.127+0000 7f8e417fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:39.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.127+0000 7f8e417fa640 10 monclient(hunting): dump: 2024-03-25T19:40:39.126 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:39.126 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:39.126 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:39.126 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:39.127 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:39.127 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:39.127 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:39.127 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:39.127 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:39.127 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:39.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.134+0000 7f8e417fa640 1 -- 172.21.15.44:0/1599373760 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3588193612 0 0) 0x7f8e441d4540 con 0x7f8e4413f710 2024-03-25T19:40:39.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.134+0000 7f8e417fa640 1 -- 172.21.15.44:0/1599373760 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e441d83d0 con 0x7f8e4413f710 2024-03-25T19:40:39.133 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.134+0000 7f8e417fa640 1 -- 172.21.15.44:0/1599373760 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f8e30001b60 con 0x7f8e4413f710 2024-03-25T19:40:39.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.134+0000 7f8e417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:39.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.134+0000 7f8e417fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:39.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.134+0000 7f8e417fa640 10 monclient(hunting): dump: 2024-03-25T19:40:39.134 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:39.134 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:39.134 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:39.134 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:39.134 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:39.134 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:39.134 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:39.134 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:39.134 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:39.134 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:39.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.140+0000 7f8e417fa640 1 -- 172.21.15.44:0/1599373760 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1476569846 0 0) 0x7f8e440abcc0 con 0x7f8e440a98f0 2024-03-25T19:40:39.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.141+0000 7f8e417fa640 1 -- 172.21.15.44:0/1599373760 >> v1:172.21.15.44:6790/0 conn(0x7f8e4413f710 legacy=0x7f8e441d4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:39.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.141+0000 7f8e417fa640 1 -- 172.21.15.44:0/1599373760 >> v1:172.21.15.44:6789/0 conn(0x7f8e441430c0 legacy=0x7f8e441383d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:39.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.141+0000 7f8e417fa640 10 monclient: _finish_hunting 0 2024-03-25T19:40:39.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.141+0000 7f8e417fa640 1 monclient: found mon.b 2024-03-25T19:40:39.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.141+0000 7f8e417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:39.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.141+0000 7f8e417fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:40:39.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.141+0000 7f8e417fa640 1 -- 172.21.15.44:0/1599373760 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e441d77a0 con 0x7f8e440a98f0 2024-03-25T19:40:39.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.141+0000 7f8e417fa640 10 monclient: _finish_auth 0 2024-03-25T19:40:39.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.141+0000 7f8e417fa640 10 monclient: _check_auth_tickets 2024-03-25T19:40:39.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.141+0000 7f8e417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:39.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.141+0000 7f8e4ad3b640 5 monclient: authenticate success, global_id 4456 2024-03-25T19:40:39.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.141+0000 7f8e4ad3b640 10 monclient: _renew_subs 2024-03-25T19:40:39.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.141+0000 7f8e4ad3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:40:39.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.141+0000 7f8e4ad3b640 1 -- 172.21.15.44:0/1599373760 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e441d41e0 con 0x7f8e440a98f0 2024-03-25T19:40:39.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.141+0000 7f8e4ad3b640 10 client.4456.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:39.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.141+0000 7f8e4ad3b640 10 monclient: _renew_subs 2024-03-25T19:40:39.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.141+0000 7f8e4ad3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:40:39.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.141+0000 7f8e4ad3b640 1 -- 172.21.15.44:0/1599373760 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e44138ae0 con 0x7f8e440a98f0 2024-03-25T19:40:39.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.141+0000 7f8e4ad3b640 1 librados: init done 2024-03-25T19:40:39.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.142+0000 7f8e4ad3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:39.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.142+0000 7f8e4ad3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:39.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.142+0000 7f8e4ad3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:40:39.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.142+0000 7f8e4ad3b640 1 -- 172.21.15.44:0/1599373760 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e44138de0 con 0x7f8e440a98f0 2024-03-25T19:40:39.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.426+0000 7f8e417fa640 1 -- 172.21.15.44:0/1599373760 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e380035f0 con 0x7f8e440a98f0 2024-03-25T19:40:39.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.426+0000 7f8e417fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:39.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.426+0000 7f8e417fa640 1 -- 172.21.15.44:0/1599373760 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f8e38003980 con 0x7f8e440a98f0 2024-03-25T19:40:39.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.426+0000 7f8e417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:39.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.426+0000 7f8e417fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:39.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.426+0000 7f8e417fa640 10 monclient: dump: 2024-03-25T19:40:39.425 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:39.425 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:39.425 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:39.425 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:39.425 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:39.426 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:39.426 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:39.426 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:39.426 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:39.426 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:39.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.426+0000 7f8e42ffd640 4 set_mon_vals no callback set 2024-03-25T19:40:39.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.427+0000 7f8e417fa640 1 -- 172.21.15.44:0/1599373760 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f8e38003b60 con 0x7f8e440a98f0 2024-03-25T19:40:39.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.427+0000 7f8e43fff640 10 monclient: get_auth_request con 0x7f8e2806e0b0 auth_method 0 2024-03-25T19:40:39.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.428+0000 7f8e417fa640 10 client.4456.objecter ms_handle_connect 0x7f8e2806e0b0 2024-03-25T19:40:39.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.432+0000 7f8e417fa640 1 -- 172.21.15.44:0/1599373760 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f8e380548c0 con 0x7f8e440a98f0 2024-03-25T19:40:39.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.432+0000 7f8e417fa640 10 client.4456.objecter ms_dispatch 0x7f8e44000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:39.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.432+0000 7f8e417fa640 3 client.4456.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:39.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.432+0000 7f8e417fa640 3 client.4456.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:39.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.432+0000 7f8e417fa640 20 client.4456.objecter dump_active .. 0 homeless 2024-03-25T19:40:39.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.432+0000 7f8e417fa640 1 -- 172.21.15.44:0/1599373760 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e380503d0 con 0x7f8e440a98f0 2024-03-25T19:40:39.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.432+0000 7f8e417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:39.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.432+0000 7f8e417fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:39.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.652+0000 7f8e4ad3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:39.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.652+0000 7f8e4ad3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:39.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.652+0000 7f8e4ad3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:40:39.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.652+0000 7f8e4ad3b640 1 -- 172.21.15.44:0/1599373760 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e44074cc0 con 0x7f8e440a98f0 2024-03-25T19:40:39.668 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:05:30.569 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:39.671 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:05:30.573 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:39.708 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:05:30.610 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:39.728 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:05:30.630 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:39.737 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.738+0000 7f8e417fa640 1 -- 172.21.15.44:0/1599373760 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f8e38052300 con 0x7f8e440a98f0 2024-03-25T19:40:39.737 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.738+0000 7f8e417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:39.737 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.738+0000 7f8e417fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:39.737 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:40:39.737 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:39.738 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:05:30.617 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:39.739 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.740+0000 7f8e4ad3b640 10 librados: watch_flush enter 2024-03-25T19:40:39.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.740+0000 7f8e4ad3b640 10 librados: watch_flush exit 2024-03-25T19:40:39.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.740+0000 7f8e4ad3b640 20 client.4456.objecter shutdown clearing up homeless session... 2024-03-25T19:40:39.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.740+0000 7f8e4ad3b640 10 client.4456.objecter successfully canceled tick 2024-03-25T19:40:39.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.740+0000 7f8e4ad3b640 1 -- 172.21.15.44:0/1599373760 >> v1:172.21.15.111:6800/3883215793 conn(0x7f8e2806e0b0 legacy=0x7f8e28070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:39.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.740+0000 7f8e4ad3b640 10 monclient: shutdown 2024-03-25T19:40:39.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.740+0000 7f8e4ad3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:39.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.740+0000 7f8e4ad3b640 1 -- 172.21.15.44:0/1599373760 >> v1:172.21.15.111:6789/0 conn(0x7f8e440a98f0 legacy=0x7f8e441d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:39.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.740+0000 7f8e4ad3b640 1 -- 172.21.15.44:0/1599373760 shutdown_connections 2024-03-25T19:40:39.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.740+0000 7f8e4ad3b640 1 -- 172.21.15.44:0/1599373760 >> 172.21.15.44:0/1599373760 conn(0x7f8e440b7ea0 msgr2=0x7f8e44074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:39.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.740+0000 7f8e4ad3b640 1 -- 172.21.15.44:0/1599373760 shutdown_connections 2024-03-25T19:40:39.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.740+0000 7f8e4ad3b640 1 -- 172.21.15.44:0/1599373760 wait complete. 2024-03-25T19:40:39.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:39.740+0000 7f8e4ad3b640 1 librados: shutdown 2024-03-25T19:40:39.741 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:05:30.642 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:39.746 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:05:30.626 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:39.750 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:39.777 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:05:30.679 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:39.778 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:05:30.657 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:46.554 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:46.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.660+0000 7f5193fc8640 10 monclient: get_monmap_and_config 2024-03-25T19:40:46.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.660+0000 7f5193fc8640 10 monclient: build_initial_monmap 2024-03-25T19:40:46.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.660+0000 7f5193fc8640 10 monclient: monmap: 2024-03-25T19:40:46.660 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:40:46.660 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:46.660 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:40:46.661404+0000 2024-03-25T19:40:46.660 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:40:46.661404+0000 2024-03-25T19:40:46.660 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:46.660 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:46.660 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:40:46.660 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:40:46.660 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:40:46.660 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:46.661 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.661+0000 7f5193fc8640 1 Processor -- start 2024-03-25T19:40:46.661 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.661+0000 7f5193fc8640 1 -- start start 2024-03-25T19:40:46.661 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.661+0000 7f5193fc8640 10 monclient: init 2024-03-25T19:40:46.662 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.661+0000 7f5193fc8640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:46.662 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.661+0000 7f5193fc8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:46.662 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.662+0000 7f5193fc8640 10 monclient(hunting): picked mon.noname-c con 0x7f518c0a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:40:46.662 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.662+0000 7f5193fc8640 10 monclient(hunting): picked mon.noname-a con 0x7f518c13f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:40:46.662 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.662+0000 7f5193fc8640 10 monclient(hunting): picked mon.noname-b con 0x7f518c1430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:40:46.662 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.662+0000 7f5193fc8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f518c142520 con 0x7f518c13f710 2024-03-25T19:40:46.662 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.662+0000 7f5193fc8640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f518c0b28a0 con 0x7f518c0a98f0 2024-03-25T19:40:46.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.662+0000 7f5193fc8640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f518c0b2a30 con 0x7f518c1430c0 2024-03-25T19:40:46.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.662+0000 7f5193fc8640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:46.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.662+0000 7f5193fc8640 10 monclient(hunting): authenticate will time out at 5710.833040s 2024-03-25T19:40:46.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.669+0000 7f5191563640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f518c13f710 0x7f518c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:37114/0 (socket says 172.21.15.44:37114) 2024-03-25T19:40:46.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.669+0000 7f5191563640 1 -- 172.21.15.44:0/1400332766 learned_addr learned my addr 172.21.15.44:0/1400332766 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:40:46.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.676+0000 7f5190d62640 1 -- 172.21.15.44:0/1400332766 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695638063 0 0) 0x7f518c142520 con 0x7f518c13f710 2024-03-25T19:40:46.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.676+0000 7f5190d62640 10 monclient(hunting): _negotiate 2024-03-25T19:40:46.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.676+0000 7f5190d62640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:46.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.676+0000 7f5190d62640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:46.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.676+0000 7f5190d62640 10 monclient(hunting): my global_id is 4476 2024-03-25T19:40:46.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.676+0000 7f5190d62640 1 -- 172.21.15.44:0/1400332766 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5178003520 con 0x7f518c13f710 2024-03-25T19:40:46.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.680+0000 7f5190d62640 1 -- 172.21.15.44:0/1400332766 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1670528605 0 0) 0x7f518c0b28a0 con 0x7f518c0a98f0 2024-03-25T19:40:46.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.680+0000 7f5190d62640 10 monclient(hunting): _negotiate 2024-03-25T19:40:46.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.680+0000 7f5190d62640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:46.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.680+0000 7f5190d62640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:46.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.680+0000 7f5190d62640 10 monclient(hunting): my global_id is 4459 2024-03-25T19:40:46.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.680+0000 7f5190d62640 1 -- 172.21.15.44:0/1400332766 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f518c142520 con 0x7f518c0a98f0 2024-03-25T19:40:46.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.690+0000 7f5190d62640 1 -- 172.21.15.44:0/1400332766 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 720959967 0 0) 0x7f5178003520 con 0x7f518c13f710 2024-03-25T19:40:46.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.690+0000 7f5190d62640 1 -- 172.21.15.44:0/1400332766 >> v1:172.21.15.44:6790/0 conn(0x7f518c1430c0 legacy=0x7f518c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:46.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.690+0000 7f5190d62640 1 -- 172.21.15.44:0/1400332766 >> v1:172.21.15.111:6789/0 conn(0x7f518c0a98f0 legacy=0x7f518c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:46.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.690+0000 7f5190d62640 10 monclient: _finish_hunting 0 2024-03-25T19:40:46.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.690+0000 7f5190d62640 1 monclient: found mon.noname-a 2024-03-25T19:40:46.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.690+0000 7f5190d62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:46.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.690+0000 7f5190d62640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:40:46.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.690+0000 7f5190d62640 1 -- 172.21.15.44:0/1400332766 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f518c0b44c0 con 0x7f518c13f710 2024-03-25T19:40:46.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.690+0000 7f5190d62640 10 monclient: _finish_auth 0 2024-03-25T19:40:46.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.690+0000 7f5190d62640 10 monclient: _check_auth_tickets 2024-03-25T19:40:46.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.690+0000 7f5190d62640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:46.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.690+0000 7f5190d62640 1 -- 172.21.15.44:0/1400332766 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f517c001690 con 0x7f518c13f710 2024-03-25T19:40:46.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.690+0000 7f5190d62640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:46.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.690+0000 7f5190d62640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:46.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.690+0000 7f5190d62640 10 monclient: dump: 2024-03-25T19:40:46.692 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:46.692 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:46.692 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:46.692 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:46.692 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:46.692 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:46.692 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:46.692 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:46.692 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:46.692 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:46.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.690+0000 7f5193fc8640 5 monclient: authenticate success, global_id 4476 2024-03-25T19:40:46.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.690+0000 7f5193fc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:46.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.709+0000 7f5190d62640 1 -- 172.21.15.44:0/1400332766 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f517c0017f0 con 0x7f518c13f710 2024-03-25T19:40:46.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.709+0000 7f5190d62640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:46.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.709+0000 7f5190d62640 1 -- 172.21.15.44:0/1400332766 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f517c001c00 con 0x7f518c13f710 2024-03-25T19:40:46.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.709+0000 7f5193fc8640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:46.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.709+0000 7f5193fc8640 4 set_mon_vals no callback set 2024-03-25T19:40:46.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.709+0000 7f5190d62640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:46.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.709+0000 7f5190d62640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:46.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.709+0000 7f5190d62640 10 monclient: dump: 2024-03-25T19:40:46.709 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:46.709 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:46.709 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:46.709 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:46.709 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:46.710 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:46.710 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:46.710 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:46.710 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:46.710 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:46.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.709+0000 7f5193fc8640 10 monclient: shutdown 2024-03-25T19:40:46.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.709+0000 7f5193fc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:46.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.710+0000 7f5193fc8640 1 -- 172.21.15.44:0/1400332766 >> v1:172.21.15.44:6789/0 conn(0x7f518c13f710 legacy=0x7f518c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:46.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.710+0000 7f5193fc8640 1 -- 172.21.15.44:0/1400332766 shutdown_connections 2024-03-25T19:40:46.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.710+0000 7f5193fc8640 1 -- 172.21.15.44:0/1400332766 >> 172.21.15.44:0/1400332766 conn(0x7f518c0b7ea0 msgr2=0x7f518c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:46.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.710+0000 7f5193fc8640 1 -- 172.21.15.44:0/1400332766 shutdown_connections 2024-03-25T19:40:46.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.710+0000 7f5193fc8640 1 -- 172.21.15.44:0/1400332766 wait complete. 2024-03-25T19:40:46.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.710+0000 7f5193fc8640 10 monclient: build_initial_monmap 2024-03-25T19:40:46.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.710+0000 7f5193fc8640 10 monclient: monmap: 2024-03-25T19:40:46.713 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:40:46.713 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:46.713 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:40:46.713 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:40:46.713 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:46.713 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:40:46.713 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:40:46.713 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:40:46.713 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:40:46.713 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:46.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.711+0000 7f5193fc8640 1 librados: starting msgr at 2024-03-25T19:40:46.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.711+0000 7f5193fc8640 1 librados: starting objecter 2024-03-25T19:40:46.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.711+0000 7f5193fc8640 1 Processor -- start 2024-03-25T19:40:46.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.711+0000 7f5193fc8640 1 -- start start 2024-03-25T19:40:46.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.711+0000 7f5193fc8640 1 librados: setting wanted keys 2024-03-25T19:40:46.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.711+0000 7f5193fc8640 1 librados: calling monclient init 2024-03-25T19:40:46.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.711+0000 7f5193fc8640 10 monclient: init 2024-03-25T19:40:46.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.712+0000 7f5193fc8640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:46.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.712+0000 7f5193fc8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:46.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.712+0000 7f5193fc8640 10 monclient(hunting): picked mon.noname-a con 0x7f518c0a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:40:46.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.712+0000 7f5193fc8640 10 monclient(hunting): picked mon.noname-b con 0x7f518c13f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:40:46.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.712+0000 7f5193fc8640 10 monclient(hunting): picked mon.noname-c con 0x7f518c1430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:40:46.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.712+0000 7f5193fc8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f518c139520 con 0x7f518c0a98f0 2024-03-25T19:40:46.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.712+0000 7f5193fc8640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f518c138b40 con 0x7f518c13f710 2024-03-25T19:40:46.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.712+0000 7f5193fc8640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f518c138e20 con 0x7f518c1430c0 2024-03-25T19:40:46.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.712+0000 7f5193fc8640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:46.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.712+0000 7f5193fc8640 10 monclient(hunting): authenticate will time out at 5710.883100s 2024-03-25T19:40:46.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.720+0000 7f5191563640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f518c13f710 0x7f518c137b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:59410/0 (socket says 172.21.15.44:59410) 2024-03-25T19:40:46.721 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.720+0000 7f5191563640 1 -- 172.21.15.44:0/1121058470 learned_addr learned my addr 172.21.15.44:0/1121058470 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:40:46.721 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.721+0000 7f51827fc640 10 client.?.objecter ms_handle_connect 0x7f518c13f710 2024-03-25T19:40:46.721 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.721+0000 7f51827fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:46.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.731+0000 7f51827fc640 10 client.?.objecter ms_handle_connect 0x7f518c1430c0 2024-03-25T19:40:46.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.731+0000 7f51827fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:46.733 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.734+0000 7f51827fc640 10 client.?.objecter ms_handle_connect 0x7f518c0a98f0 2024-03-25T19:40:46.733 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.734+0000 7f51827fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:46.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.735+0000 7f5191d64640 1 -- 172.21.15.44:0/1121058470 >> v1:172.21.15.44:6789/0 conn(0x7f518c0a98f0 legacy=0x7f518c137460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T19:40:46.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.735+0000 7f5191d64640 1 -- 172.21.15.44:0/1121058470 >> v1:172.21.15.44:6789/0 conn(0x7f518c0a98f0 legacy=0x7f518c137460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:40:46.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.735+0000 7f5191d64640 1 --1- 172.21.15.44:0/1121058470 >> v1:172.21.15.44:6789/0 conn(0x7f518c0a98f0 0x7f518c137460 :-1 s=OPENED pgs=36762 cs=1 l=1).handle_message read tag failed 2024-03-25T19:40:46.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.735+0000 7f5191d64640 1 --1- 172.21.15.44:0/1121058470 >> v1:172.21.15.44:6789/0 conn(0x7f518c0a98f0 0x7f518c137460 :-1 s=OPENED pgs=36762 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:40:46.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.735+0000 7f51827fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.44:6789/0 2024-03-25T19:40:46.737 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.737+0000 7f51827fc640 1 -- 172.21.15.44:0/1121058470 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117588589 0 0) 0x7f518c138e20 con 0x7f518c1430c0 2024-03-25T19:40:46.737 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.737+0000 7f51827fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:46.737 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.737+0000 7f51827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:46.738 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.737+0000 7f51827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:46.738 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.737+0000 7f51827fc640 10 monclient(hunting): my global_id is 4451 2024-03-25T19:40:46.738 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.737+0000 7f51827fc640 1 -- 172.21.15.44:0/1121058470 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5170004eb0 con 0x7f518c1430c0 2024-03-25T19:40:46.745 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.745+0000 7f51827fc640 1 -- 172.21.15.44:0/1121058470 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3484451989 0 0) 0x7f518c138b40 con 0x7f518c13f710 2024-03-25T19:40:46.745 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.745+0000 7f51827fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:46.745 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.745+0000 7f51827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:46.745 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.745+0000 7f51827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:46.745 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.745+0000 7f51827fc640 10 monclient(hunting): my global_id is 4462 2024-03-25T19:40:46.745 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.745+0000 7f51827fc640 1 -- 172.21.15.44:0/1121058470 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f518c138e20 con 0x7f518c13f710 2024-03-25T19:40:46.748 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.748+0000 7f51827fc640 1 -- 172.21.15.44:0/1121058470 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1478897489 0 0) 0x7f5170004eb0 con 0x7f518c1430c0 2024-03-25T19:40:46.748 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.748+0000 7f51827fc640 1 -- 172.21.15.44:0/1121058470 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f518c138b40 con 0x7f518c1430c0 2024-03-25T19:40:46.748 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.748+0000 7f51827fc640 1 -- 172.21.15.44:0/1121058470 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f5184001b20 con 0x7f518c1430c0 2024-03-25T19:40:46.748 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.748+0000 7f51827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:46.748 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.748+0000 7f51827fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:46.748 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.748+0000 7f51827fc640 10 monclient(hunting): dump: 2024-03-25T19:40:46.748 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:46.748 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:46.748 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:46.749 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:46.749 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:46.749 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:46.749 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:46.749 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:46.749 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:46.749 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:46.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.748+0000 7f51827fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:46.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.769+0000 7f51827fc640 1 -- 172.21.15.44:0/1121058470 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 98885506 0 0) 0x7f518c138b40 con 0x7f518c1430c0 2024-03-25T19:40:46.770 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.769+0000 7f51827fc640 1 -- 172.21.15.44:0/1121058470 >> v1:172.21.15.111:6789/0 conn(0x7f518c13f710 legacy=0x7f518c137b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:46.770 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.769+0000 7f51827fc640 1 -- 172.21.15.44:0/1121058470 >> v1:172.21.15.44:6789/0 conn(0x7f518c0a98f0 legacy=0x7f518c137460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:40:46.770 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.769+0000 7f51827fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:46.770 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.769+0000 7f51827fc640 1 monclient: found mon.c 2024-03-25T19:40:46.770 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.769+0000 7f51827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:46.770 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.769+0000 7f51827fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:40:46.770 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.769+0000 7f51827fc640 1 -- 172.21.15.44:0/1121058470 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f518c139100 con 0x7f518c1430c0 2024-03-25T19:40:46.770 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.769+0000 7f51827fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:46.770 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.769+0000 7f51827fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:46.770 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.769+0000 7f51827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:46.770 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.769+0000 7f5193fc8640 5 monclient: authenticate success, global_id 4451 2024-03-25T19:40:46.770 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.769+0000 7f5193fc8640 10 monclient: _renew_subs 2024-03-25T19:40:46.770 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.769+0000 7f5193fc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:40:46.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.769+0000 7f5193fc8640 1 -- 172.21.15.44:0/1121058470 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f518c136aa0 con 0x7f518c1430c0 2024-03-25T19:40:46.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.769+0000 7f5193fc8640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:46.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.769+0000 7f5193fc8640 10 monclient: _renew_subs 2024-03-25T19:40:46.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.769+0000 7f5193fc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:40:46.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.769+0000 7f5193fc8640 1 -- 172.21.15.44:0/1121058470 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f518c13ba10 con 0x7f518c1430c0 2024-03-25T19:40:46.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.769+0000 7f5193fc8640 1 librados: init done 2024-03-25T19:40:46.772 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.770+0000 7f5193fc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:46.772 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.770+0000 7f5193fc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:46.772 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.770+0000 7f5193fc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:40:46.772 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.770+0000 7f5193fc8640 1 -- 172.21.15.44:0/1121058470 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f518c0627b0 con 0x7f518c1430c0 2024-03-25T19:40:46.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.775+0000 7f51827fc640 1 -- 172.21.15.44:0/1121058470 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5184002030 con 0x7f518c1430c0 2024-03-25T19:40:46.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.775+0000 7f51827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:46.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:46.775+0000 7f5183fff640 4 set_mon_vals no callback set 2024-03-25T19:40:47.093 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.093+0000 7f51827fc640 1 -- 172.21.15.44:0/1121058470 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f5184003b10 con 0x7f518c1430c0 2024-03-25T19:40:47.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.093+0000 7f51827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:47.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.093+0000 7f51827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:47.094 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.093+0000 7f51827fc640 10 monclient: dump: 2024-03-25T19:40:47.094 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:47.094 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:47.094 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:47.094 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:47.094 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:47.094 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:47.094 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:47.094 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:47.094 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:47.094 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:47.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.094+0000 7f51827fc640 1 -- 172.21.15.44:0/1121058470 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f5184003cf0 con 0x7f518c1430c0 2024-03-25T19:40:47.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.095+0000 7f5191d64640 10 monclient: get_auth_request con 0x7f517006dca0 auth_method 0 2024-03-25T19:40:47.095 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.095+0000 7f51827fc640 10 client.4451.objecter ms_handle_connect 0x7f517006dca0 2024-03-25T19:40:47.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.103+0000 7f51827fc640 1 -- 172.21.15.44:0/1121058470 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f51840537f0 con 0x7f518c1430c0 2024-03-25T19:40:47.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.103+0000 7f51827fc640 10 client.4451.objecter ms_dispatch 0x7f518c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:47.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.103+0000 7f51827fc640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:47.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.103+0000 7f51827fc640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:47.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.103+0000 7f51827fc640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-25T19:40:47.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.103+0000 7f51827fc640 1 -- 172.21.15.44:0/1121058470 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f518404f300 con 0x7f518c1430c0 2024-03-25T19:40:47.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.103+0000 7f51827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:47.103 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.103+0000 7f51827fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:47.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.357+0000 7f5193fc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:47.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.357+0000 7f5193fc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:47.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.357+0000 7f5193fc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:40:47.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.357+0000 7f5193fc8640 1 -- 172.21.15.44:0/1121058470 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f518c13df90 con 0x7f518c1430c0 2024-03-25T19:40:47.388 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.388+0000 7f51827fc640 1 -- 172.21.15.44:0/1121058470 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f5184051230 con 0x7f518c1430c0 2024-03-25T19:40:47.388 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.388+0000 7f51827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:47.388 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.388+0000 7f51827fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:47.388 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:40:47.389 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:47.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.390+0000 7f5193fc8640 10 librados: watch_flush enter 2024-03-25T19:40:47.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.390+0000 7f5193fc8640 10 librados: watch_flush exit 2024-03-25T19:40:47.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.390+0000 7f5193fc8640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-25T19:40:47.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.390+0000 7f5193fc8640 10 client.4451.objecter successfully canceled tick 2024-03-25T19:40:47.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.391+0000 7f5193fc8640 1 -- 172.21.15.44:0/1121058470 >> v1:172.21.15.111:6800/3883215793 conn(0x7f517006dca0 legacy=0x7f5170070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:47.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.391+0000 7f5193fc8640 10 monclient: shutdown 2024-03-25T19:40:47.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.391+0000 7f5193fc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:47.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.391+0000 7f5193fc8640 1 -- 172.21.15.44:0/1121058470 >> v1:172.21.15.44:6790/0 conn(0x7f518c1430c0 legacy=0x7f518c1e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:47.392 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.391+0000 7f5193fc8640 1 -- 172.21.15.44:0/1121058470 shutdown_connections 2024-03-25T19:40:47.392 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.391+0000 7f5193fc8640 1 -- 172.21.15.44:0/1121058470 >> 172.21.15.44:0/1121058470 conn(0x7f518c0b7ea0 msgr2=0x7f518c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:47.392 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.391+0000 7f5193fc8640 1 -- 172.21.15.44:0/1121058470 shutdown_connections 2024-03-25T19:40:47.392 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.391+0000 7f5193fc8640 1 -- 172.21.15.44:0/1121058470 wait complete. 2024-03-25T19:40:47.392 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:47.391+0000 7f5193fc8640 1 librados: shutdown 2024-03-25T19:40:47.402 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:54.209 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:54.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.321+0000 7fb1cd127640 10 monclient: get_monmap_and_config 2024-03-25T19:40:54.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.321+0000 7fb1cd127640 10 monclient: build_initial_monmap 2024-03-25T19:40:54.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.321+0000 7fb1cd127640 10 monclient: monmap: 2024-03-25T19:40:54.321 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:40:54.322 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:54.322 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:40:54.322807+0000 2024-03-25T19:40:54.322 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:40:54.322807+0000 2024-03-25T19:40:54.322 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:54.322 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:54.322 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:40:54.322 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:40:54.322 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:40:54.322 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:54.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.322+0000 7fb1cd127640 1 Processor -- start 2024-03-25T19:40:54.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.322+0000 7fb1cd127640 1 -- start start 2024-03-25T19:40:54.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.322+0000 7fb1cd127640 10 monclient: init 2024-03-25T19:40:54.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.323+0000 7fb1cd127640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:54.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.323+0000 7fb1cd127640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:54.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.323+0000 7fb1cd127640 10 monclient(hunting): picked mon.noname-c con 0x7fb1c80b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T19:40:54.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.323+0000 7fb1cd127640 10 monclient(hunting): picked mon.noname-b con 0x7fb1c80b1730 addr v1:172.21.15.44:6790/0 2024-03-25T19:40:54.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.323+0000 7fb1cd127640 10 monclient(hunting): picked mon.noname-a con 0x7fb1c814be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:40:54.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.323+0000 7fb1cd127640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c80af940 con 0x7fb1c814be90 2024-03-25T19:40:54.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.323+0000 7fb1cd127640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c80b0a70 con 0x7fb1c80b4d60 2024-03-25T19:40:54.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.323+0000 7fb1cd127640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c80b0d50 con 0x7fb1c80b1730 2024-03-25T19:40:54.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.323+0000 7fb1cd127640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:54.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.323+0000 7fb1cd127640 10 monclient(hunting): authenticate will time out at 5718.494458s 2024-03-25T19:40:54.337 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.337+0000 7fb1c6d9d640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb1c80b4d60 0x7fb1c813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:58032/0 (socket says 172.21.15.44:58032) 2024-03-25T19:40:54.337 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.337+0000 7fb1c6d9d640 1 -- 172.21.15.44:0/1712494820 learned_addr learned my addr 172.21.15.44:0/1712494820 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:40:54.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.351+0000 7fb1c5d9b640 1 -- 172.21.15.44:0/1712494820 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321220321 0 0) 0x7fb1c80b0a70 con 0x7fb1c80b4d60 2024-03-25T19:40:54.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.351+0000 7fb1c5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:40:54.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.351+0000 7fb1c5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:54.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.351+0000 7fb1c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:54.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.351+0000 7fb1c5d9b640 10 monclient(hunting): my global_id is 4465 2024-03-25T19:40:54.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.351+0000 7fb1c5d9b640 1 -- 172.21.15.44:0/1712494820 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1b0003520 con 0x7fb1c80b4d60 2024-03-25T19:40:54.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.358+0000 7fb1c5d9b640 1 -- 172.21.15.44:0/1712494820 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4243817720 0 0) 0x7fb1b0003520 con 0x7fb1c80b4d60 2024-03-25T19:40:54.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.358+0000 7fb1c5d9b640 1 -- 172.21.15.44:0/1712494820 >> v1:172.21.15.44:6790/0 conn(0x7fb1c80b1730 legacy=0x7fb1c80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:54.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.358+0000 7fb1c5d9b640 1 -- 172.21.15.44:0/1712494820 >> v1:172.21.15.44:6789/0 conn(0x7fb1c814be90 legacy=0x7fb1c814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:54.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.358+0000 7fb1c5d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:40:54.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.358+0000 7fb1c5d9b640 1 monclient: found mon.noname-c 2024-03-25T19:40:54.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.358+0000 7fb1c5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:54.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.358+0000 7fb1c5d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:40:54.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.358+0000 7fb1c5d9b640 1 -- 172.21.15.44:0/1712494820 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c80b1030 con 0x7fb1c80b4d60 2024-03-25T19:40:54.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.358+0000 7fb1c5d9b640 10 monclient: _finish_auth 0 2024-03-25T19:40:54.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.358+0000 7fb1c5d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:40:54.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.358+0000 7fb1c5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:54.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.358+0000 7fb1c5d9b640 1 -- 172.21.15.44:0/1712494820 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fb1b8001690 con 0x7fb1c80b4d60 2024-03-25T19:40:54.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.358+0000 7fb1c5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:54.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.358+0000 7fb1c5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:54.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.358+0000 7fb1c5d9b640 10 monclient: dump: 2024-03-25T19:40:54.359 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:54.359 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:54.360 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:54.360 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:54.360 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:54.360 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:54.360 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:54.360 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:54.360 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:54.360 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:54.360 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.358+0000 7fb1cd127640 5 monclient: authenticate success, global_id 4465 2024-03-25T19:40:54.360 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.358+0000 7fb1cd127640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:54.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.369+0000 7fb1c5d9b640 1 -- 172.21.15.44:0/1712494820 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1b80017f0 con 0x7fb1c80b4d60 2024-03-25T19:40:54.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.369+0000 7fb1c5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:54.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.369+0000 7fb1c5d9b640 1 -- 172.21.15.44:0/1712494820 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fb1b8001c40 con 0x7fb1c80b4d60 2024-03-25T19:40:54.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.369+0000 7fb1c5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:54.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.369+0000 7fb1c5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:54.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.369+0000 7fb1c5d9b640 10 monclient: dump: 2024-03-25T19:40:54.369 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:54.369 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:54.369 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:54.369 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:54.370 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:54.370 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:54.370 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:54.370 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:54.370 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:54.370 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:54.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.369+0000 7fb1cd127640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:54.370 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.369+0000 7fb1cd127640 4 set_mon_vals no callback set 2024-03-25T19:40:54.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.369+0000 7fb1cd127640 10 monclient: shutdown 2024-03-25T19:40:54.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.369+0000 7fb1cd127640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:54.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.369+0000 7fb1cd127640 1 -- 172.21.15.44:0/1712494820 >> v1:172.21.15.111:6789/0 conn(0x7fb1c80b4d60 legacy=0x7fb1c813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:54.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.370+0000 7fb1cd127640 1 -- 172.21.15.44:0/1712494820 shutdown_connections 2024-03-25T19:40:54.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.370+0000 7fb1cd127640 1 -- 172.21.15.44:0/1712494820 >> 172.21.15.44:0/1712494820 conn(0x7fb1c80617a0 msgr2=0x7fb1c8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:54.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.370+0000 7fb1cd127640 1 -- 172.21.15.44:0/1712494820 shutdown_connections 2024-03-25T19:40:54.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.370+0000 7fb1cd127640 1 -- 172.21.15.44:0/1712494820 wait complete. 2024-03-25T19:40:54.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.370+0000 7fb1cd127640 10 monclient: build_initial_monmap 2024-03-25T19:40:54.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.370+0000 7fb1cd127640 10 monclient: monmap: 2024-03-25T19:40:54.372 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:40:54.372 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:54.372 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:40:54.372 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:40:54.372 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:54.372 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:40:54.372 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:40:54.372 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:40:54.372 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:40:54.373 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:54.373 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.371+0000 7fb1cd127640 1 librados: starting msgr at 2024-03-25T19:40:54.373 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.371+0000 7fb1cd127640 1 librados: starting objecter 2024-03-25T19:40:54.373 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.371+0000 7fb1cd127640 1 Processor -- start 2024-03-25T19:40:54.373 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.371+0000 7fb1cd127640 1 -- start start 2024-03-25T19:40:54.373 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.371+0000 7fb1cd127640 1 librados: setting wanted keys 2024-03-25T19:40:54.373 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.371+0000 7fb1cd127640 1 librados: calling monclient init 2024-03-25T19:40:54.373 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.371+0000 7fb1cd127640 10 monclient: init 2024-03-25T19:40:54.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.372+0000 7fb1cd127640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:54.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.372+0000 7fb1cd127640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:54.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.372+0000 7fb1cd127640 10 monclient(hunting): picked mon.noname-c con 0x7fb1c80b1730 addr v1:172.21.15.44:6790/0 2024-03-25T19:40:54.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.372+0000 7fb1cd127640 10 monclient(hunting): picked mon.noname-b con 0x7fb1c80b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T19:40:54.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.372+0000 7fb1cd127640 10 monclient(hunting): picked mon.noname-a con 0x7fb1c814be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:40:54.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.372+0000 7fb1cd127640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c8141f40 con 0x7fb1c814be90 2024-03-25T19:40:54.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.372+0000 7fb1cd127640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c8142360 con 0x7fb1c80b4d60 2024-03-25T19:40:54.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.372+0000 7fb1cd127640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c81424f0 con 0x7fb1c80b1730 2024-03-25T19:40:54.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.372+0000 7fb1cd127640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:54.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.372+0000 7fb1cd127640 10 monclient(hunting): authenticate will time out at 5718.543342s 2024-03-25T19:40:54.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.374+0000 7fb1c759e640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fb1c814be90 0x7fb1c81e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:54480/0 (socket says 172.21.15.44:54480) 2024-03-25T19:40:54.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.374+0000 7fb1c759e640 1 -- 172.21.15.44:0/1909855135 learned_addr learned my addr 172.21.15.44:0/1909855135 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:40:54.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.375+0000 7fb1af7fe640 10 client.?.objecter ms_handle_connect 0x7fb1c814be90 2024-03-25T19:40:54.376 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.375+0000 7fb1af7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:54.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.378+0000 7fb1af7fe640 10 client.?.objecter ms_handle_connect 0x7fb1c80b1730 2024-03-25T19:40:54.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.378+0000 7fb1af7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:54.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.378+0000 7fb1af7fe640 10 client.?.objecter ms_handle_connect 0x7fb1c80b4d60 2024-03-25T19:40:54.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.378+0000 7fb1af7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:54.385 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.385+0000 7fb1af7fe640 1 -- 172.21.15.44:0/1909855135 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33615087 0 0) 0x7fb1c8142360 con 0x7fb1c80b4d60 2024-03-25T19:40:54.385 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.385+0000 7fb1af7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:54.385 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.385+0000 7fb1af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:54.385 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.385+0000 7fb1af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:54.385 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.385+0000 7fb1af7fe640 10 monclient(hunting): my global_id is 4468 2024-03-25T19:40:54.385 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.385+0000 7fb1af7fe640 1 -- 172.21.15.44:0/1909855135 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1b4004eb0 con 0x7fb1c80b4d60 2024-03-25T19:40:54.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.391+0000 7fb1af7fe640 1 -- 172.21.15.44:0/1909855135 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2968520752 0 0) 0x7fb1c8141f40 con 0x7fb1c814be90 2024-03-25T19:40:54.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.391+0000 7fb1af7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:54.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.391+0000 7fb1af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:54.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.391+0000 7fb1af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:54.392 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.391+0000 7fb1af7fe640 10 monclient(hunting): my global_id is 4482 2024-03-25T19:40:54.392 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.391+0000 7fb1af7fe640 1 -- 172.21.15.44:0/1909855135 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1c8142360 con 0x7fb1c814be90 2024-03-25T19:40:54.392 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.392+0000 7fb1af7fe640 1 -- 172.21.15.44:0/1909855135 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408146010 0 0) 0x7fb1c80b1030 con 0x7fb1c80b1730 2024-03-25T19:40:54.392 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.392+0000 7fb1af7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:54.392 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.392+0000 7fb1af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:54.392 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.392+0000 7fb1af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:54.392 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.392+0000 7fb1af7fe640 10 monclient(hunting): my global_id is 4457 2024-03-25T19:40:54.392 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.392+0000 7fb1af7fe640 1 -- 172.21.15.44:0/1909855135 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1c8141f40 con 0x7fb1c80b1730 2024-03-25T19:40:54.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.406+0000 7fb1af7fe640 1 -- 172.21.15.44:0/1909855135 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3939826419 0 0) 0x7fb1b4004eb0 con 0x7fb1c80b4d60 2024-03-25T19:40:54.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.406+0000 7fb1af7fe640 1 -- 172.21.15.44:0/1909855135 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1c80b1030 con 0x7fb1c80b4d60 2024-03-25T19:40:54.407 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.406+0000 7fb1af7fe640 1 -- 172.21.15.44:0/1909855135 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fb1bc0018c0 con 0x7fb1c80b4d60 2024-03-25T19:40:54.407 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.406+0000 7fb1af7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:54.407 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.406+0000 7fb1af7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:54.407 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.406+0000 7fb1af7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:54.407 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:54.407 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:54.407 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:54.407 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:54.407 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:54.407 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:54.407 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:54.407 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:54.407 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:54.407 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:54.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.406+0000 7fb1af7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:54.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.408+0000 7fb1af7fe640 1 -- 172.21.15.44:0/1909855135 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2584023686 0 0) 0x7fb1c8141f40 con 0x7fb1c80b1730 2024-03-25T19:40:54.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.408+0000 7fb1af7fe640 1 -- 172.21.15.44:0/1909855135 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1b4004eb0 con 0x7fb1c80b1730 2024-03-25T19:40:54.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.408+0000 7fb1af7fe640 1 -- 172.21.15.44:0/1909855135 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fb1b8001960 con 0x7fb1c80b1730 2024-03-25T19:40:54.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.408+0000 7fb1af7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:54.408 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.408+0000 7fb1af7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:54.409 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.408+0000 7fb1af7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:54.409 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:54.409 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:54.409 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:54.409 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:54.409 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:54.409 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:54.409 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:54.409 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:54.409 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:54.409 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:54.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.415+0000 7fb1af7fe640 1 -- 172.21.15.44:0/1909855135 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3143646723 0 0) 0x7fb1c80b1030 con 0x7fb1c80b4d60 2024-03-25T19:40:54.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.415+0000 7fb1af7fe640 1 -- 172.21.15.44:0/1909855135 >> v1:172.21.15.44:6790/0 conn(0x7fb1c80b1730 legacy=0x7fb1c81405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:54.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.415+0000 7fb1af7fe640 1 -- 172.21.15.44:0/1909855135 >> v1:172.21.15.44:6789/0 conn(0x7fb1c814be90 legacy=0x7fb1c81e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:54.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.415+0000 7fb1af7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:54.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.415+0000 7fb1af7fe640 1 monclient: found mon.b 2024-03-25T19:40:54.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.415+0000 7fb1af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:54.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.415+0000 7fb1af7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:40:54.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.415+0000 7fb1af7fe640 1 -- 172.21.15.44:0/1909855135 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c8140d00 con 0x7fb1c80b4d60 2024-03-25T19:40:54.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.415+0000 7fb1af7fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:54.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.415+0000 7fb1af7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:54.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.415+0000 7fb1af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:54.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.415+0000 7fb1cd127640 5 monclient: authenticate success, global_id 4468 2024-03-25T19:40:54.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.415+0000 7fb1cd127640 10 monclient: _renew_subs 2024-03-25T19:40:54.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.415+0000 7fb1cd127640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:40:54.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.415+0000 7fb1cd127640 1 -- 172.21.15.44:0/1909855135 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1c8138150 con 0x7fb1c80b4d60 2024-03-25T19:40:54.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.415+0000 7fb1cd127640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:54.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.415+0000 7fb1cd127640 10 monclient: _renew_subs 2024-03-25T19:40:54.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.415+0000 7fb1cd127640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:40:54.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.415+0000 7fb1cd127640 1 -- 172.21.15.44:0/1909855135 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1c81d8520 con 0x7fb1c80b4d60 2024-03-25T19:40:54.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.415+0000 7fb1cd127640 1 librados: init done 2024-03-25T19:40:54.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.416+0000 7fb1cd127640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:54.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.416+0000 7fb1cd127640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:54.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.416+0000 7fb1cd127640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:40:54.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.416+0000 7fb1cd127640 1 -- 172.21.15.44:0/1909855135 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1c8138450 con 0x7fb1c80b4d60 2024-03-25T19:40:54.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.440+0000 7fb1af7fe640 1 -- 172.21.15.44:0/1909855135 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1bc0035f0 con 0x7fb1c80b4d60 2024-03-25T19:40:54.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.441+0000 7fb1af7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:54.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.441+0000 7fb1af7fe640 1 -- 172.21.15.44:0/1909855135 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fb1bc003a00 con 0x7fb1c80b4d60 2024-03-25T19:40:54.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.441+0000 7fb1af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:54.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.441+0000 7fb1af7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:54.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.441+0000 7fb1af7fe640 10 monclient: dump: 2024-03-25T19:40:54.441 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:40:54.441 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:40:54.441 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:54.441 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:40:54.441 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:54.442 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:40:54.442 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:40:54.442 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:40:54.442 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:40:54.442 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:40:54.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.441+0000 7fb1c559a640 4 set_mon_vals no callback set 2024-03-25T19:40:54.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.442+0000 7fb1af7fe640 1 -- 172.21.15.44:0/1909855135 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7fb1bc003c20 con 0x7fb1c80b4d60 2024-03-25T19:40:54.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.443+0000 7fb1c6d9d640 10 monclient: get_auth_request con 0x7fb1b406df70 auth_method 0 2024-03-25T19:40:54.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.443+0000 7fb1af7fe640 10 client.4468.objecter ms_handle_connect 0x7fb1b406df70 2024-03-25T19:40:54.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.447+0000 7fb1af7fe640 1 -- 172.21.15.44:0/1909855135 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7fb1bc053530 con 0x7fb1c80b4d60 2024-03-25T19:40:54.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.447+0000 7fb1af7fe640 10 client.4468.objecter ms_dispatch 0x7fb1c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:54.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.447+0000 7fb1af7fe640 3 client.4468.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:54.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.447+0000 7fb1af7fe640 3 client.4468.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:54.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.447+0000 7fb1af7fe640 20 client.4468.objecter dump_active .. 0 homeless 2024-03-25T19:40:54.710 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.710+0000 7fb1af7fe640 1 -- 172.21.15.44:0/1909855135 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb1bc04f040 con 0x7fb1c80b4d60 2024-03-25T19:40:54.710 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.710+0000 7fb1af7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:54.710 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.710+0000 7fb1af7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:54.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.945+0000 7fb1cd127640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:54.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.945+0000 7fb1cd127640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:54.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.945+0000 7fb1cd127640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:40:54.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.945+0000 7fb1cd127640 1 -- 172.21.15.44:0/1909855135 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1c80627b0 con 0x7fb1c80b4d60 2024-03-25T19:40:54.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.967+0000 7fb1af7fe640 1 -- 172.21.15.44:0/1909855135 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7fb1bc050f70 con 0x7fb1c80b4d60 2024-03-25T19:40:54.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.967+0000 7fb1af7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:54.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.967+0000 7fb1af7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:54.967 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:40:54.967 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:54.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.969+0000 7fb1cd127640 10 librados: watch_flush enter 2024-03-25T19:40:54.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.969+0000 7fb1cd127640 10 librados: watch_flush exit 2024-03-25T19:40:54.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.969+0000 7fb1cd127640 20 client.4468.objecter shutdown clearing up homeless session... 2024-03-25T19:40:54.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.969+0000 7fb1cd127640 10 client.4468.objecter successfully canceled tick 2024-03-25T19:40:54.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.969+0000 7fb1cd127640 1 -- 172.21.15.44:0/1909855135 >> v1:172.21.15.111:6800/3883215793 conn(0x7fb1b406df70 legacy=0x7fb1b4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:54.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.969+0000 7fb1cd127640 10 monclient: shutdown 2024-03-25T19:40:54.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.969+0000 7fb1cd127640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:54.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.969+0000 7fb1cd127640 1 -- 172.21.15.44:0/1909855135 >> v1:172.21.15.111:6789/0 conn(0x7fb1c80b4d60 legacy=0x7fb1c81d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:54.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.969+0000 7fb1cd127640 1 -- 172.21.15.44:0/1909855135 shutdown_connections 2024-03-25T19:40:54.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.969+0000 7fb1cd127640 1 -- 172.21.15.44:0/1909855135 >> 172.21.15.44:0/1909855135 conn(0x7fb1c80617a0 msgr2=0x7fb1c80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:54.971 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.970+0000 7fb1cd127640 1 -- 172.21.15.44:0/1909855135 shutdown_connections 2024-03-25T19:40:54.971 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.970+0000 7fb1cd127640 1 -- 172.21.15.44:0/1909855135 wait complete. 2024-03-25T19:40:54.971 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:40:54.970+0000 7fb1cd127640 1 librados: shutdown 2024-03-25T19:40:54.981 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:59.625 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:05:50.526 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:59.632 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:05:50.534 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:59.664 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:05:50.543 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:59.671 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:05:50.550 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:59.674 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:05:50.576 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:59.677 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:05:50.578 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:59.685 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:05:50.587 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:59.707 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:05:50.586 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:59.713 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:05:50.615 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:01.788 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:01.902 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.902+0000 7f2718855640 10 monclient: get_monmap_and_config 2024-03-25T19:41:01.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.902+0000 7f2718855640 10 monclient: build_initial_monmap 2024-03-25T19:41:01.903 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.902+0000 7f2718855640 10 monclient: monmap: 2024-03-25T19:41:01.903 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:41:01.903 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:01.903 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:41:01.903464+0000 2024-03-25T19:41:01.903 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:41:01.903464+0000 2024-03-25T19:41:01.903 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:01.903 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:01.903 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:41:01.903 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:41:01.903 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:41:01.903 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:01.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.902+0000 7f2718855640 1 Processor -- start 2024-03-25T19:41:01.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.903+0000 7f2718855640 1 -- start start 2024-03-25T19:41:01.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.903+0000 7f2718855640 10 monclient: init 2024-03-25T19:41:01.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.903+0000 7f2718855640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:01.904 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.903+0000 7f2718855640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:01.905 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.904+0000 7f2718855640 10 monclient(hunting): picked mon.noname-a con 0x7f27140a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:41:01.905 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.904+0000 7f2718855640 10 monclient(hunting): picked mon.noname-c con 0x7f271413f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:41:01.905 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.904+0000 7f2718855640 10 monclient(hunting): picked mon.noname-b con 0x7f27141430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:41:01.905 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.904+0000 7f2718855640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2714142520 con 0x7f27140a98f0 2024-03-25T19:41:01.905 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.904+0000 7f2718855640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27140b28a0 con 0x7f271413f710 2024-03-25T19:41:01.905 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.904+0000 7f2718855640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27140b2a30 con 0x7f27141430c0 2024-03-25T19:41:01.905 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.904+0000 7f2718855640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:01.905 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.904+0000 7f2718855640 10 monclient(hunting): authenticate will time out at 5726.074909s 2024-03-25T19:41:01.919 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.919+0000 7f271259c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f27140a98f0 0x7f27140a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:38988/0 (socket says 172.21.15.44:38988) 2024-03-25T19:41:01.919 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.919+0000 7f271259c640 1 -- 172.21.15.44:0/3352858443 learned_addr learned my addr 172.21.15.44:0/3352858443 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:41:01.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.938+0000 7f271159a640 1 -- 172.21.15.44:0/3352858443 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061787797 0 0) 0x7f27140b28a0 con 0x7f271413f710 2024-03-25T19:41:01.939 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.938+0000 7f271159a640 10 monclient(hunting): _negotiate 2024-03-25T19:41:01.939 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.938+0000 7f271159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:01.939 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.938+0000 7f271159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:01.939 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.938+0000 7f271159a640 10 monclient(hunting): my global_id is 4471 2024-03-25T19:41:01.939 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.938+0000 7f271159a640 1 -- 172.21.15.44:0/3352858443 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26fc003520 con 0x7f271413f710 2024-03-25T19:41:01.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.940+0000 7f271159a640 1 -- 172.21.15.44:0/3352858443 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2473626443 0 0) 0x7f2714142520 con 0x7f27140a98f0 2024-03-25T19:41:01.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.940+0000 7f271159a640 10 monclient(hunting): _negotiate 2024-03-25T19:41:01.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.940+0000 7f271159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:01.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.940+0000 7f271159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:01.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.940+0000 7f271159a640 10 monclient(hunting): my global_id is 4485 2024-03-25T19:41:01.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.940+0000 7f271159a640 1 -- 172.21.15.44:0/3352858443 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27140b28a0 con 0x7f27140a98f0 2024-03-25T19:41:01.941 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.941+0000 7f271159a640 1 -- 172.21.15.44:0/3352858443 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2876378370 0 0) 0x7f27140b2a30 con 0x7f27141430c0 2024-03-25T19:41:01.941 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.941+0000 7f271159a640 10 monclient(hunting): _negotiate 2024-03-25T19:41:01.941 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.941+0000 7f271159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:01.941 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.941+0000 7f271159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:01.941 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.941+0000 7f271159a640 10 monclient(hunting): my global_id is 4460 2024-03-25T19:41:01.941 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.941+0000 7f271159a640 1 -- 172.21.15.44:0/3352858443 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2714142520 con 0x7f27141430c0 2024-03-25T19:41:01.957 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.956+0000 7f271159a640 1 -- 172.21.15.44:0/3352858443 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3277634618 0 0) 0x7f2714142520 con 0x7f27141430c0 2024-03-25T19:41:01.957 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.957+0000 7f271159a640 1 -- 172.21.15.44:0/3352858443 >> v1:172.21.15.111:6789/0 conn(0x7f271413f710 legacy=0x7f27141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:01.957 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.957+0000 7f271159a640 1 -- 172.21.15.44:0/3352858443 >> v1:172.21.15.44:6789/0 conn(0x7f27140a98f0 legacy=0x7f27140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:01.957 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.957+0000 7f271159a640 10 monclient: _finish_hunting 0 2024-03-25T19:41:01.957 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.957+0000 7f271159a640 1 monclient: found mon.noname-b 2024-03-25T19:41:01.957 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.957+0000 7f271159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:01.957 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.957+0000 7f271159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:41:01.957 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.957+0000 7f271159a640 1 -- 172.21.15.44:0/3352858443 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27140b44c0 con 0x7f27141430c0 2024-03-25T19:41:01.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.957+0000 7f271159a640 10 monclient: _finish_auth 0 2024-03-25T19:41:01.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.957+0000 7f271159a640 10 monclient: _check_auth_tickets 2024-03-25T19:41:01.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.957+0000 7f271159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:01.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.957+0000 7f271159a640 1 -- 172.21.15.44:0/3352858443 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f270c001930 con 0x7f27141430c0 2024-03-25T19:41:01.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.957+0000 7f271159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:01.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.957+0000 7f271159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:01.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.957+0000 7f271159a640 10 monclient: dump: 2024-03-25T19:41:01.958 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:01.958 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:01.958 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:01.958 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:01.958 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:01.958 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:01.959 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:01.959 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:01.959 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:01.959 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:01.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.957+0000 7f2718855640 5 monclient: authenticate success, global_id 4460 2024-03-25T19:41:01.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.957+0000 7f2718855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:01.973 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.973+0000 7f271159a640 1 -- 172.21.15.44:0/3352858443 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f270c001a90 con 0x7f27141430c0 2024-03-25T19:41:01.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.973+0000 7f271159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:01.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.973+0000 7f271159a640 1 -- 172.21.15.44:0/3352858443 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f270c001ee0 con 0x7f27141430c0 2024-03-25T19:41:01.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.973+0000 7f2718855640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:01.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.973+0000 7f2718855640 4 set_mon_vals no callback set 2024-03-25T19:41:01.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.973+0000 7f271159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:01.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.973+0000 7f271159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:01.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.973+0000 7f271159a640 10 monclient: dump: 2024-03-25T19:41:01.974 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:01.974 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:01.974 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:01.974 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:01.975 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:01.975 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:01.975 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:01.975 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:01.975 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:01.975 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:01.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.974+0000 7f2718855640 10 monclient: shutdown 2024-03-25T19:41:01.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.974+0000 7f2718855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:01.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.974+0000 7f2718855640 1 -- 172.21.15.44:0/3352858443 >> v1:172.21.15.44:6790/0 conn(0x7f27141430c0 legacy=0x7f271414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:01.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.975+0000 7f2718855640 1 -- 172.21.15.44:0/3352858443 shutdown_connections 2024-03-25T19:41:01.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.975+0000 7f2718855640 1 -- 172.21.15.44:0/3352858443 >> 172.21.15.44:0/3352858443 conn(0x7f27140b7ea0 msgr2=0x7f27140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:01.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.975+0000 7f2718855640 1 -- 172.21.15.44:0/3352858443 shutdown_connections 2024-03-25T19:41:01.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.975+0000 7f2718855640 1 -- 172.21.15.44:0/3352858443 wait complete. 2024-03-25T19:41:01.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.976+0000 7f2718855640 10 monclient: build_initial_monmap 2024-03-25T19:41:01.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.976+0000 7f2718855640 10 monclient: monmap: 2024-03-25T19:41:01.977 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:41:01.977 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:01.977 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:41:01.977 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:41:01.977 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:01.977 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:41:01.978 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:41:01.978 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:41:01.978 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:41:01.978 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:01.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.976+0000 7f2718855640 1 librados: starting msgr at 2024-03-25T19:41:01.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.976+0000 7f2718855640 1 librados: starting objecter 2024-03-25T19:41:01.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.976+0000 7f2718855640 1 Processor -- start 2024-03-25T19:41:01.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.976+0000 7f2718855640 1 -- start start 2024-03-25T19:41:01.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.976+0000 7f2718855640 1 librados: setting wanted keys 2024-03-25T19:41:01.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.976+0000 7f2718855640 1 librados: calling monclient init 2024-03-25T19:41:01.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.976+0000 7f2718855640 10 monclient: init 2024-03-25T19:41:01.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.977+0000 7f2718855640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:01.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.977+0000 7f2718855640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:01.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.977+0000 7f2718855640 10 monclient(hunting): picked mon.noname-b con 0x7f27140a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:41:01.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.977+0000 7f2718855640 10 monclient(hunting): picked mon.noname-c con 0x7f271413f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:41:01.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.977+0000 7f2718855640 10 monclient(hunting): picked mon.noname-a con 0x7f27141430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:41:01.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.977+0000 7f2718855640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27140b0090 con 0x7f27141430c0 2024-03-25T19:41:01.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.977+0000 7f2718855640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27140b0d60 con 0x7f27140a98f0 2024-03-25T19:41:01.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.977+0000 7f2718855640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27140b1040 con 0x7f271413f710 2024-03-25T19:41:01.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.977+0000 7f2718855640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:01.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.977+0000 7f2718855640 10 monclient(hunting): authenticate will time out at 5726.148472s 2024-03-25T19:41:01.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.980+0000 7f271259c640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f27140a98f0 0x7f27140ad440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:56372/0 (socket says 172.21.15.44:56372) 2024-03-25T19:41:01.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.980+0000 7f271259c640 1 -- 172.21.15.44:0/1682563477 learned_addr learned my addr 172.21.15.44:0/1682563477 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:41:01.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.981+0000 7f2702ffd640 10 client.?.objecter ms_handle_connect 0x7f27140a98f0 2024-03-25T19:41:01.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.981+0000 7f2702ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:01.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.989+0000 7f2702ffd640 1 -- 172.21.15.44:0/1682563477 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2801257786 0 0) 0x7f27140b0d60 con 0x7f27140a98f0 2024-03-25T19:41:01.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.989+0000 7f2702ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:01.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.989+0000 7f2702ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:01.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.989+0000 7f2702ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:01.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.989+0000 7f2702ffd640 10 monclient(hunting): my global_id is 4474 2024-03-25T19:41:01.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.989+0000 7f2702ffd640 1 -- 172.21.15.44:0/1682563477 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26f4004eb0 con 0x7f27140a98f0 2024-03-25T19:41:01.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.994+0000 7f2702ffd640 1 -- 172.21.15.44:0/1682563477 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3682907271 0 0) 0x7f26f4004eb0 con 0x7f27140a98f0 2024-03-25T19:41:01.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.994+0000 7f2702ffd640 1 -- 172.21.15.44:0/1682563477 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27140b0d60 con 0x7f27140a98f0 2024-03-25T19:41:01.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.994+0000 7f2702ffd640 1 -- 172.21.15.44:0/1682563477 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f27040019b0 con 0x7f27140a98f0 2024-03-25T19:41:01.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.994+0000 7f2702ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:01.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.994+0000 7f2702ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:01.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.994+0000 7f2702ffd640 10 monclient(hunting): dump: 2024-03-25T19:41:01.995 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:01.995 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:01.995 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:01.995 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:01.995 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:01.995 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:01.995 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:01.995 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:01.997 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:01.997 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:01.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.994+0000 7f2702ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:01.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.995+0000 7f2702ffd640 10 client.?.objecter ms_handle_connect 0x7f271413f710 2024-03-25T19:41:01.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.995+0000 7f2702ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:01.999 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.999+0000 7f2702ffd640 10 client.?.objecter ms_handle_connect 0x7f27141430c0 2024-03-25T19:41:01.999 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:01.999+0000 7f2702ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:02.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.006+0000 7f2702ffd640 1 -- 172.21.15.44:0/1682563477 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517281374 0 0) 0x7f27140b1040 con 0x7f271413f710 2024-03-25T19:41:02.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.006+0000 7f2702ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:02.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.006+0000 7f2702ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:02.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.006+0000 7f2702ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:02.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.006+0000 7f2702ffd640 10 monclient(hunting): my global_id is 4463 2024-03-25T19:41:02.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.006+0000 7f2702ffd640 1 -- 172.21.15.44:0/1682563477 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26f4004eb0 con 0x7f271413f710 2024-03-25T19:41:02.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.009+0000 7f2702ffd640 1 -- 172.21.15.44:0/1682563477 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3830325778 0 0) 0x7f27140b0d60 con 0x7f27140a98f0 2024-03-25T19:41:02.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.009+0000 7f2702ffd640 1 -- 172.21.15.44:0/1682563477 >> v1:172.21.15.44:6790/0 conn(0x7f271413f710 legacy=0x7f27140af730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:02.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.009+0000 7f2702ffd640 1 -- 172.21.15.44:0/1682563477 >> v1:172.21.15.44:6789/0 conn(0x7f27141430c0 legacy=0x7f27141e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:02.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.009+0000 7f2702ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:02.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.009+0000 7f2702ffd640 1 monclient: found mon.b 2024-03-25T19:41:02.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.009+0000 7f2702ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:02.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.009+0000 7f2702ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:41:02.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.009+0000 7f2702ffd640 1 -- 172.21.15.44:0/1682563477 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27140b1320 con 0x7f27140a98f0 2024-03-25T19:41:02.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.009+0000 7f2702ffd640 10 monclient: _finish_auth 0 2024-03-25T19:41:02.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.009+0000 7f2702ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:02.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.009+0000 7f2702ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:02.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.009+0000 7f2718855640 5 monclient: authenticate success, global_id 4474 2024-03-25T19:41:02.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.009+0000 7f2718855640 10 monclient: _renew_subs 2024-03-25T19:41:02.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.009+0000 7f2718855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:41:02.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.009+0000 7f2718855640 1 -- 172.21.15.44:0/1682563477 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27140aecc0 con 0x7f27140a98f0 2024-03-25T19:41:02.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.009+0000 7f2718855640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:02.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.009+0000 7f2718855640 10 monclient: _renew_subs 2024-03-25T19:41:02.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.009+0000 7f2718855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:41:02.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.009+0000 7f2718855640 1 -- 172.21.15.44:0/1682563477 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27141438d0 con 0x7f27140a98f0 2024-03-25T19:41:02.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.009+0000 7f2718855640 1 librados: init done 2024-03-25T19:41:02.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.010+0000 7f2718855640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:02.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.010+0000 7f2718855640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:02.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.010+0000 7f2718855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:41:02.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.010+0000 7f2718855640 1 -- 172.21.15.44:0/1682563477 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27140b3ef0 con 0x7f27140a98f0 2024-03-25T19:41:02.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.276+0000 7f2702ffd640 1 -- 172.21.15.44:0/1682563477 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27040047e0 con 0x7f27140a98f0 2024-03-25T19:41:02.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.276+0000 7f2702ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:02.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.276+0000 7f2702ffd640 1 -- 172.21.15.44:0/1682563477 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f2704004bf0 con 0x7f27140a98f0 2024-03-25T19:41:02.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.276+0000 7f2702ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:02.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.276+0000 7f2702ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:02.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.276+0000 7f2702ffd640 10 monclient: dump: 2024-03-25T19:41:02.276 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:02.277 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:02.277 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:02.277 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:02.277 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:02.277 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:02.277 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:02.277 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:02.277 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:02.277 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:02.277 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.276+0000 7f2703fff640 4 set_mon_vals no callback set 2024-03-25T19:41:02.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.277+0000 7f2702ffd640 1 -- 172.21.15.44:0/1682563477 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f2704004e10 con 0x7f27140a98f0 2024-03-25T19:41:02.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.277+0000 7f2711d9b640 10 monclient: get_auth_request con 0x7f26f406dd40 auth_method 0 2024-03-25T19:41:02.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.277+0000 7f2702ffd640 10 client.4474.objecter ms_handle_connect 0x7f26f406dd40 2024-03-25T19:41:02.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.282+0000 7f2702ffd640 1 -- 172.21.15.44:0/1682563477 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f2704054970 con 0x7f27140a98f0 2024-03-25T19:41:02.282 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.282+0000 7f2702ffd640 10 client.4474.objecter ms_dispatch 0x7f2714000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:02.282 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.282+0000 7f2702ffd640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:02.282 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.282+0000 7f2702ffd640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:02.282 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.282+0000 7f2702ffd640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-25T19:41:02.282 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.282+0000 7f2702ffd640 1 -- 172.21.15.44:0/1682563477 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2704050480 con 0x7f27140a98f0 2024-03-25T19:41:02.282 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.282+0000 7f2702ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:02.282 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.282+0000 7f2702ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:02.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.516+0000 7f2718855640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:02.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.516+0000 7f2718855640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:02.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.516+0000 7f2718855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:41:02.517 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.516+0000 7f2718855640 1 -- 172.21.15.44:0/1682563477 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27140afe40 con 0x7f27140a98f0 2024-03-25T19:41:02.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.582+0000 7f2702ffd640 1 -- 172.21.15.44:0/1682563477 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f27040523b0 con 0x7f27140a98f0 2024-03-25T19:41:02.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.582+0000 7f2702ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:02.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.582+0000 7f2702ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:02.583 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:41:02.583 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:02.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.584+0000 7f2718855640 10 librados: watch_flush enter 2024-03-25T19:41:02.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.585+0000 7f2718855640 10 librados: watch_flush exit 2024-03-25T19:41:02.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.585+0000 7f2718855640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-25T19:41:02.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.585+0000 7f2718855640 10 client.4474.objecter successfully canceled tick 2024-03-25T19:41:02.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.585+0000 7f2718855640 1 -- 172.21.15.44:0/1682563477 >> v1:172.21.15.111:6800/3883215793 conn(0x7f26f406dd40 legacy=0x7f26f4070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:02.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.585+0000 7f2718855640 10 monclient: shutdown 2024-03-25T19:41:02.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.585+0000 7f2718855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:02.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.585+0000 7f2718855640 1 -- 172.21.15.44:0/1682563477 >> v1:172.21.15.111:6789/0 conn(0x7f27140a98f0 legacy=0x7f27140ad440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:02.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.585+0000 7f2718855640 1 -- 172.21.15.44:0/1682563477 shutdown_connections 2024-03-25T19:41:02.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.585+0000 7f2718855640 1 -- 172.21.15.44:0/1682563477 >> 172.21.15.44:0/1682563477 conn(0x7f27140b7ea0 msgr2=0x7f2714074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:02.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.585+0000 7f2718855640 1 -- 172.21.15.44:0/1682563477 shutdown_connections 2024-03-25T19:41:02.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.585+0000 7f2718855640 1 -- 172.21.15.44:0/1682563477 wait complete. 2024-03-25T19:41:02.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:02.585+0000 7f2718855640 1 librados: shutdown 2024-03-25T19:41:02.596 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:09.403 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:09.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.510+0000 7fe4fb3ab640 10 monclient: get_monmap_and_config 2024-03-25T19:41:09.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.510+0000 7fe4fb3ab640 10 monclient: build_initial_monmap 2024-03-25T19:41:09.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.510+0000 7fe4fb3ab640 10 monclient: monmap: 2024-03-25T19:41:09.510 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:41:09.510 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:09.510 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:41:09.511480+0000 2024-03-25T19:41:09.510 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:41:09.511480+0000 2024-03-25T19:41:09.511 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:09.511 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:09.511 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:41:09.511 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:41:09.511 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:41:09.511 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:09.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.511+0000 7fe4fb3ab640 1 Processor -- start 2024-03-25T19:41:09.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.511+0000 7fe4fb3ab640 1 -- start start 2024-03-25T19:41:09.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.511+0000 7fe4fb3ab640 10 monclient: init 2024-03-25T19:41:09.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.511+0000 7fe4fb3ab640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:09.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.511+0000 7fe4fb3ab640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:09.512 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.511+0000 7fe4fb3ab640 10 monclient(hunting): picked mon.noname-c con 0x7fe4f40b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T19:41:09.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.511+0000 7fe4fb3ab640 10 monclient(hunting): picked mon.noname-b con 0x7fe4f40b1730 addr v1:172.21.15.44:6790/0 2024-03-25T19:41:09.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.511+0000 7fe4fb3ab640 10 monclient(hunting): picked mon.noname-a con 0x7fe4f414be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:41:09.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.511+0000 7fe4fb3ab640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f40af940 con 0x7fe4f414be90 2024-03-25T19:41:09.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.511+0000 7fe4fb3ab640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f40b0a70 con 0x7fe4f40b4d60 2024-03-25T19:41:09.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.511+0000 7fe4fb3ab640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f40b0d50 con 0x7fe4f40b1730 2024-03-25T19:41:09.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.511+0000 7fe4fb3ab640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:09.513 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.511+0000 7fe4fb3ab640 10 monclient(hunting): authenticate will time out at 5733.682736s 2024-03-25T19:41:09.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.519+0000 7fe4f9147640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe4f40b4d60 0x7fe4f413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:52952/0 (socket says 172.21.15.44:52952) 2024-03-25T19:41:09.519 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.519+0000 7fe4f9147640 1 -- 172.21.15.44:0/3484545970 learned_addr learned my addr 172.21.15.44:0/3484545970 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:41:09.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.538+0000 7fe4ebfff640 1 -- 172.21.15.44:0/3484545970 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3132903689 0 0) 0x7fe4f40b0a70 con 0x7fe4f40b4d60 2024-03-25T19:41:09.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.538+0000 7fe4ebfff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:09.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.538+0000 7fe4ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:09.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.538+0000 7fe4ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:09.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.538+0000 7fe4ebfff640 10 monclient(hunting): my global_id is 4477 2024-03-25T19:41:09.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.538+0000 7fe4ebfff640 1 -- 172.21.15.44:0/3484545970 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4e0003520 con 0x7fe4f40b4d60 2024-03-25T19:41:09.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.546+0000 7fe4ebfff640 1 -- 172.21.15.44:0/3484545970 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3842521406 0 0) 0x7fe4e0003520 con 0x7fe4f40b4d60 2024-03-25T19:41:09.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.546+0000 7fe4ebfff640 1 -- 172.21.15.44:0/3484545970 >> v1:172.21.15.44:6790/0 conn(0x7fe4f40b1730 legacy=0x7fe4f40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:09.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.546+0000 7fe4ebfff640 1 -- 172.21.15.44:0/3484545970 >> v1:172.21.15.44:6789/0 conn(0x7fe4f414be90 legacy=0x7fe4f414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:09.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.546+0000 7fe4ebfff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:09.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.546+0000 7fe4ebfff640 1 monclient: found mon.noname-c 2024-03-25T19:41:09.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.546+0000 7fe4ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:09.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.546+0000 7fe4ebfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:41:09.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.546+0000 7fe4ebfff640 1 -- 172.21.15.44:0/3484545970 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f40b1030 con 0x7fe4f40b4d60 2024-03-25T19:41:09.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.546+0000 7fe4ebfff640 10 monclient: _finish_auth 0 2024-03-25T19:41:09.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.546+0000 7fe4ebfff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:09.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.546+0000 7fe4ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:09.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.546+0000 7fe4ebfff640 1 -- 172.21.15.44:0/3484545970 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fe4f0001690 con 0x7fe4f40b4d60 2024-03-25T19:41:09.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.546+0000 7fe4ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:09.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.546+0000 7fe4ebfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:09.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.546+0000 7fe4ebfff640 10 monclient: dump: 2024-03-25T19:41:09.548 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:09.548 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:09.548 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:09.548 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:09.548 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:09.548 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:09.548 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:09.548 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:09.548 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:09.548 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:09.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.546+0000 7fe4fb3ab640 5 monclient: authenticate success, global_id 4477 2024-03-25T19:41:09.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.546+0000 7fe4fb3ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:09.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.555+0000 7fe4ebfff640 1 -- 172.21.15.44:0/3484545970 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4f00017f0 con 0x7fe4f40b4d60 2024-03-25T19:41:09.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.556+0000 7fe4ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:09.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.556+0000 7fe4ebfff640 1 -- 172.21.15.44:0/3484545970 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fe4f0001c00 con 0x7fe4f40b4d60 2024-03-25T19:41:09.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.556+0000 7fe4fb3ab640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:09.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.556+0000 7fe4fb3ab640 4 set_mon_vals no callback set 2024-03-25T19:41:09.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.556+0000 7fe4ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:09.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.556+0000 7fe4ebfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:09.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.556+0000 7fe4ebfff640 10 monclient: dump: 2024-03-25T19:41:09.556 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:09.556 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:09.556 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:09.556 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:09.556 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:09.556 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:09.556 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:09.556 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:09.557 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:09.557 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:09.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.556+0000 7fe4fb3ab640 10 monclient: shutdown 2024-03-25T19:41:09.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.556+0000 7fe4fb3ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:09.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.556+0000 7fe4fb3ab640 1 -- 172.21.15.44:0/3484545970 >> v1:172.21.15.111:6789/0 conn(0x7fe4f40b4d60 legacy=0x7fe4f413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:09.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.556+0000 7fe4fb3ab640 1 -- 172.21.15.44:0/3484545970 shutdown_connections 2024-03-25T19:41:09.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.556+0000 7fe4fb3ab640 1 -- 172.21.15.44:0/3484545970 >> 172.21.15.44:0/3484545970 conn(0x7fe4f40617a0 msgr2=0x7fe4f4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:09.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.556+0000 7fe4fb3ab640 1 -- 172.21.15.44:0/3484545970 shutdown_connections 2024-03-25T19:41:09.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.556+0000 7fe4fb3ab640 1 -- 172.21.15.44:0/3484545970 wait complete. 2024-03-25T19:41:09.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.557+0000 7fe4fb3ab640 10 monclient: build_initial_monmap 2024-03-25T19:41:09.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.557+0000 7fe4fb3ab640 10 monclient: monmap: 2024-03-25T19:41:09.560 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:41:09.560 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:09.560 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:41:09.560 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:41:09.560 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:09.560 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:41:09.560 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:41:09.560 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:41:09.560 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:41:09.560 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:09.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.557+0000 7fe4fb3ab640 1 librados: starting msgr at 2024-03-25T19:41:09.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.557+0000 7fe4fb3ab640 1 librados: starting objecter 2024-03-25T19:41:09.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.557+0000 7fe4fb3ab640 1 Processor -- start 2024-03-25T19:41:09.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.557+0000 7fe4fb3ab640 1 -- start start 2024-03-25T19:41:09.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.557+0000 7fe4fb3ab640 1 librados: setting wanted keys 2024-03-25T19:41:09.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.557+0000 7fe4fb3ab640 1 librados: calling monclient init 2024-03-25T19:41:09.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.557+0000 7fe4fb3ab640 10 monclient: init 2024-03-25T19:41:09.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.558+0000 7fe4fb3ab640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:09.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.558+0000 7fe4fb3ab640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:09.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.558+0000 7fe4fb3ab640 10 monclient(hunting): picked mon.noname-c con 0x7fe4f40b1730 addr v1:172.21.15.44:6790/0 2024-03-25T19:41:09.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.558+0000 7fe4fb3ab640 10 monclient(hunting): picked mon.noname-b con 0x7fe4f40b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T19:41:09.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.558+0000 7fe4fb3ab640 10 monclient(hunting): picked mon.noname-a con 0x7fe4f414be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:41:09.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.558+0000 7fe4fb3ab640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f4140d00 con 0x7fe4f414be90 2024-03-25T19:41:09.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.558+0000 7fe4fb3ab640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f4141cc0 con 0x7fe4f40b4d60 2024-03-25T19:41:09.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.558+0000 7fe4fb3ab640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f41d8470 con 0x7fe4f40b1730 2024-03-25T19:41:09.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.558+0000 7fe4fb3ab640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:09.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.558+0000 7fe4fb3ab640 10 monclient(hunting): authenticate will time out at 5733.729484s 2024-03-25T19:41:09.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.569+0000 7fe4f9948640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fe4f414be90 0x7fe4f4134d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:34040/0 (socket says 172.21.15.44:34040) 2024-03-25T19:41:09.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.569+0000 7fe4f9948640 1 -- 172.21.15.44:0/211395331 learned_addr learned my addr 172.21.15.44:0/211395331 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:41:09.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.573+0000 7fe4e9ffb640 10 client.?.objecter ms_handle_connect 0x7fe4f40b1730 2024-03-25T19:41:09.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.573+0000 7fe4e9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:09.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.575+0000 7fe4e9ffb640 10 client.?.objecter ms_handle_connect 0x7fe4f414be90 2024-03-25T19:41:09.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.575+0000 7fe4e9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:09.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.580+0000 7fe4e9ffb640 10 client.?.objecter ms_handle_connect 0x7fe4f40b4d60 2024-03-25T19:41:09.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.580+0000 7fe4e9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:09.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.586+0000 7fe4e9ffb640 1 -- 172.21.15.44:0/211395331 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1326348657 0 0) 0x7fe4f41d8470 con 0x7fe4f40b1730 2024-03-25T19:41:09.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.586+0000 7fe4e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:09.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.586+0000 7fe4e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:09.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.586+0000 7fe4e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:09.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.586+0000 7fe4e9ffb640 10 monclient(hunting): my global_id is 4469 2024-03-25T19:41:09.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.586+0000 7fe4e9ffb640 1 -- 172.21.15.44:0/211395331 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d8004eb0 con 0x7fe4f40b1730 2024-03-25T19:41:09.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.588+0000 7fe4e9ffb640 1 -- 172.21.15.44:0/211395331 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535940316 0 0) 0x7fe4f4140d00 con 0x7fe4f414be90 2024-03-25T19:41:09.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.588+0000 7fe4e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:09.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.588+0000 7fe4e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:09.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.588+0000 7fe4e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:09.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.588+0000 7fe4e9ffb640 10 monclient(hunting): my global_id is 4494 2024-03-25T19:41:09.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.588+0000 7fe4e9ffb640 1 -- 172.21.15.44:0/211395331 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4f41d8470 con 0x7fe4f414be90 2024-03-25T19:41:09.598 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.598+0000 7fe4e9ffb640 1 -- 172.21.15.44:0/211395331 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3573189955 0 0) 0x7fe4f4141cc0 con 0x7fe4f40b4d60 2024-03-25T19:41:09.598 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.598+0000 7fe4e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:09.598 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.598+0000 7fe4e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:09.598 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.598+0000 7fe4e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:09.598 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.598+0000 7fe4e9ffb640 10 monclient(hunting): my global_id is 4480 2024-03-25T19:41:09.598 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.598+0000 7fe4e9ffb640 1 -- 172.21.15.44:0/211395331 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4f4140d00 con 0x7fe4f40b4d60 2024-03-25T19:41:09.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.602+0000 7fe4e9ffb640 1 -- 172.21.15.44:0/211395331 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2117485738 0 0) 0x7fe4f41d8470 con 0x7fe4f414be90 2024-03-25T19:41:09.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.602+0000 7fe4e9ffb640 1 -- 172.21.15.44:0/211395331 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4f4141cc0 con 0x7fe4f414be90 2024-03-25T19:41:09.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.613+0000 7fe4e9ffb640 1 -- 172.21.15.44:0/211395331 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 271046180 0 0) 0x7fe4d8004eb0 con 0x7fe4f40b1730 2024-03-25T19:41:09.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.613+0000 7fe4e9ffb640 1 -- 172.21.15.44:0/211395331 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4f41d8470 con 0x7fe4f40b1730 2024-03-25T19:41:09.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.613+0000 7fe4e9ffb640 1 -- 172.21.15.44:0/211395331 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fe4f0001bc0 con 0x7fe4f40b1730 2024-03-25T19:41:09.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.613+0000 7fe4e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:09.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.613+0000 7fe4e9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:09.613 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.613+0000 7fe4e9ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:09.613 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:09.614 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:09.614 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:09.614 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:09.614 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:09.614 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:09.614 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:09.614 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:09.614 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:09.614 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:09.614 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.613+0000 7fe4e9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:09.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.614+0000 7fe4e9ffb640 1 -- 172.21.15.44:0/211395331 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fe4ec001cb0 con 0x7fe4f414be90 2024-03-25T19:41:09.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.614+0000 7fe4e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:09.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.614+0000 7fe4e9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:09.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.614+0000 7fe4e9ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:09.615 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:09.615 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:09.615 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:09.615 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:09.615 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:09.615 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:09.615 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:09.615 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:09.615 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:09.616 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:09.617 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.617+0000 7fe4e9ffb640 1 -- 172.21.15.44:0/211395331 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2247608975 0 0) 0x7fe4f4141cc0 con 0x7fe4f414be90 2024-03-25T19:41:09.617 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.617+0000 7fe4e9ffb640 1 -- 172.21.15.44:0/211395331 >> v1:172.21.15.44:6790/0 conn(0x7fe4f40b1730 legacy=0x7fe4f41405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:09.617 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.617+0000 7fe4e9ffb640 1 -- 172.21.15.44:0/211395331 >> v1:172.21.15.111:6789/0 conn(0x7fe4f40b4d60 legacy=0x7fe4f41d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:09.617 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.617+0000 7fe4e9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:41:09.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.617+0000 7fe4e9ffb640 1 monclient: found mon.a 2024-03-25T19:41:09.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.617+0000 7fe4e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:09.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.617+0000 7fe4e9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:41:09.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.617+0000 7fe4e9ffb640 1 -- 172.21.15.44:0/211395331 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f41d8600 con 0x7fe4f414be90 2024-03-25T19:41:09.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.617+0000 7fe4e9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:41:09.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.617+0000 7fe4e9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:41:09.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.617+0000 7fe4e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:09.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.617+0000 7fe4fb3ab640 5 monclient: authenticate success, global_id 4494 2024-03-25T19:41:09.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.617+0000 7fe4fb3ab640 10 monclient: _renew_subs 2024-03-25T19:41:09.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.617+0000 7fe4fb3ab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:41:09.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.617+0000 7fe4fb3ab640 1 -- 172.21.15.44:0/211395331 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f413b3c0 con 0x7fe4f414be90 2024-03-25T19:41:09.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.617+0000 7fe4fb3ab640 10 client.4494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:09.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.617+0000 7fe4fb3ab640 10 monclient: _renew_subs 2024-03-25T19:41:09.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.617+0000 7fe4fb3ab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:41:09.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.617+0000 7fe4fb3ab640 1 -- 172.21.15.44:0/211395331 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f41425a0 con 0x7fe4f414be90 2024-03-25T19:41:09.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.617+0000 7fe4fb3ab640 1 librados: init done 2024-03-25T19:41:09.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.618+0000 7fe4fb3ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:09.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.618+0000 7fe4fb3ab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:09.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.618+0000 7fe4fb3ab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:41:09.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.618+0000 7fe4fb3ab640 1 -- 172.21.15.44:0/211395331 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f40b2270 con 0x7fe4f414be90 2024-03-25T19:41:09.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.940+0000 7fe4e9ffb640 1 -- 172.21.15.44:0/211395331 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4ec003990 con 0x7fe4f414be90 2024-03-25T19:41:09.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.940+0000 7fe4e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:09.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.940+0000 7fe4e9ffb640 1 -- 172.21.15.44:0/211395331 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fe4ec003da0 con 0x7fe4f414be90 2024-03-25T19:41:09.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.940+0000 7fe4e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:09.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.940+0000 7fe4e9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:09.941 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.940+0000 7fe4e9ffb640 10 monclient: dump: 2024-03-25T19:41:09.941 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:09.941 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:09.941 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:09.941 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:09.941 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:09.941 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:09.941 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:09.941 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:09.941 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:09.941 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:09.941 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.940+0000 7fe4eb7fe640 4 set_mon_vals no callback set 2024-03-25T19:41:09.942 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.941+0000 7fe4e9ffb640 1 -- 172.21.15.44:0/211395331 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7fe4ec003fc0 con 0x7fe4f414be90 2024-03-25T19:41:09.942 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.941+0000 7fe4f9147640 10 monclient: get_auth_request con 0x7fe4d806dfc0 auth_method 0 2024-03-25T19:41:09.942 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.942+0000 7fe4e9ffb640 10 client.4494.objecter ms_handle_connect 0x7fe4d806dfc0 2024-03-25T19:41:09.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.947+0000 7fe4e9ffb640 1 -- 172.21.15.44:0/211395331 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7fe4ec0549d0 con 0x7fe4f414be90 2024-03-25T19:41:09.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.947+0000 7fe4e9ffb640 10 client.4494.objecter ms_dispatch 0x7fe4f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:09.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.947+0000 7fe4e9ffb640 3 client.4494.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:09.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.947+0000 7fe4e9ffb640 3 client.4494.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:09.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.947+0000 7fe4e9ffb640 20 client.4494.objecter dump_active .. 0 homeless 2024-03-25T19:41:09.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.947+0000 7fe4e9ffb640 1 -- 172.21.15.44:0/211395331 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe4ec0504e0 con 0x7fe4f414be90 2024-03-25T19:41:09.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.947+0000 7fe4e9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:09.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:09.947+0000 7fe4e9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:10.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:10.205+0000 7fe4fb3ab640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:10.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:10.205+0000 7fe4fb3ab640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:10.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:10.205+0000 7fe4fb3ab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:41:10.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:10.205+0000 7fe4fb3ab640 1 -- 172.21.15.44:0/211395331 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4f40627b0 con 0x7fe4f414be90 2024-03-25T19:41:10.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:10.241+0000 7fe4e9ffb640 1 -- 172.21.15.44:0/211395331 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7fe4ec052410 con 0x7fe4f414be90 2024-03-25T19:41:10.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:10.241+0000 7fe4e9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:10.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:10.241+0000 7fe4e9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:10.241 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:41:10.242 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:10.243 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:10.243+0000 7fe4fb3ab640 10 librados: watch_flush enter 2024-03-25T19:41:10.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:10.243+0000 7fe4fb3ab640 10 librados: watch_flush exit 2024-03-25T19:41:10.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:10.243+0000 7fe4fb3ab640 20 client.4494.objecter shutdown clearing up homeless session... 2024-03-25T19:41:10.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:10.243+0000 7fe4fb3ab640 10 client.4494.objecter successfully canceled tick 2024-03-25T19:41:10.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:10.243+0000 7fe4fb3ab640 1 -- 172.21.15.44:0/211395331 >> v1:172.21.15.111:6800/3883215793 conn(0x7fe4d806dfc0 legacy=0x7fe4d8070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:10.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:10.243+0000 7fe4fb3ab640 10 monclient: shutdown 2024-03-25T19:41:10.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:10.243+0000 7fe4fb3ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:10.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:10.243+0000 7fe4fb3ab640 1 -- 172.21.15.44:0/211395331 >> v1:172.21.15.44:6789/0 conn(0x7fe4f414be90 legacy=0x7fe4f4134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:10.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:10.243+0000 7fe4fb3ab640 1 -- 172.21.15.44:0/211395331 shutdown_connections 2024-03-25T19:41:10.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:10.243+0000 7fe4fb3ab640 1 -- 172.21.15.44:0/211395331 >> 172.21.15.44:0/211395331 conn(0x7fe4f40617a0 msgr2=0x7fe4f40b6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:10.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:10.243+0000 7fe4fb3ab640 1 -- 172.21.15.44:0/211395331 shutdown_connections 2024-03-25T19:41:10.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:10.243+0000 7fe4fb3ab640 1 -- 172.21.15.44:0/211395331 wait complete. 2024-03-25T19:41:10.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:10.243+0000 7fe4fb3ab640 1 librados: shutdown 2024-03-25T19:41:10.256 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:17.062 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:17.178 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.178+0000 7fe762cfb640 10 monclient: get_monmap_and_config 2024-03-25T19:41:17.179 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.178+0000 7fe762cfb640 10 monclient: build_initial_monmap 2024-03-25T19:41:17.179 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.178+0000 7fe762cfb640 10 monclient: monmap: 2024-03-25T19:41:17.179 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:41:17.179 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:17.179 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:41:17.179891+0000 2024-03-25T19:41:17.179 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:41:17.179891+0000 2024-03-25T19:41:17.179 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:17.179 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:17.179 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:41:17.179 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:41:17.179 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:41:17.179 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:17.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.179+0000 7fe762cfb640 1 Processor -- start 2024-03-25T19:41:17.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.179+0000 7fe762cfb640 1 -- start start 2024-03-25T19:41:17.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.179+0000 7fe762cfb640 10 monclient: init 2024-03-25T19:41:17.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.180+0000 7fe762cfb640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:17.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.180+0000 7fe762cfb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:17.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.180+0000 7fe762cfb640 10 monclient(hunting): picked mon.noname-a con 0x7fe75c0a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:41:17.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.180+0000 7fe762cfb640 10 monclient(hunting): picked mon.noname-b con 0x7fe75c13f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:41:17.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.180+0000 7fe762cfb640 10 monclient(hunting): picked mon.noname-c con 0x7fe75c1430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:41:17.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.180+0000 7fe762cfb640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe75c142520 con 0x7fe75c0a98f0 2024-03-25T19:41:17.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.180+0000 7fe762cfb640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe75c0b28a0 con 0x7fe75c1430c0 2024-03-25T19:41:17.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.180+0000 7fe762cfb640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe75c0b2a30 con 0x7fe75c13f710 2024-03-25T19:41:17.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.180+0000 7fe762cfb640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:17.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.180+0000 7fe762cfb640 10 monclient(hunting): authenticate will time out at 5741.351393s 2024-03-25T19:41:17.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.189+0000 7fe761298640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe75c1430c0 0x7fe75c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:57822/0 (socket says 172.21.15.44:57822) 2024-03-25T19:41:17.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.189+0000 7fe761298640 1 -- 172.21.15.44:0/180770881 learned_addr learned my addr 172.21.15.44:0/180770881 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:41:17.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.201+0000 7fe7537fe640 1 -- 172.21.15.44:0/180770881 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514045040 0 0) 0x7fe75c0b28a0 con 0x7fe75c1430c0 2024-03-25T19:41:17.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.201+0000 7fe7537fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:17.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.201+0000 7fe7537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:17.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.201+0000 7fe7537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:17.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.201+0000 7fe7537fe640 10 monclient(hunting): my global_id is 4483 2024-03-25T19:41:17.201 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.201+0000 7fe7537fe640 1 -- 172.21.15.44:0/180770881 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe744003520 con 0x7fe75c1430c0 2024-03-25T19:41:17.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.210+0000 7fe7537fe640 1 -- 172.21.15.44:0/180770881 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3435735892 0 0) 0x7fe744003520 con 0x7fe75c1430c0 2024-03-25T19:41:17.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.210+0000 7fe7537fe640 1 -- 172.21.15.44:0/180770881 >> v1:172.21.15.44:6790/0 conn(0x7fe75c13f710 legacy=0x7fe75c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:17.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.210+0000 7fe7537fe640 1 -- 172.21.15.44:0/180770881 >> v1:172.21.15.44:6789/0 conn(0x7fe75c0a98f0 legacy=0x7fe75c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:17.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.210+0000 7fe7537fe640 10 monclient: _finish_hunting 0 2024-03-25T19:41:17.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.210+0000 7fe7537fe640 1 monclient: found mon.noname-c 2024-03-25T19:41:17.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.210+0000 7fe7537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:17.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.210+0000 7fe7537fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:41:17.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.210+0000 7fe7537fe640 1 -- 172.21.15.44:0/180770881 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75c0b44c0 con 0x7fe75c1430c0 2024-03-25T19:41:17.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.210+0000 7fe7537fe640 10 monclient: _finish_auth 0 2024-03-25T19:41:17.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.210+0000 7fe7537fe640 10 monclient: _check_auth_tickets 2024-03-25T19:41:17.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.210+0000 7fe7537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:17.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.210+0000 7fe7537fe640 1 -- 172.21.15.44:0/180770881 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fe754001a10 con 0x7fe75c1430c0 2024-03-25T19:41:17.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.210+0000 7fe7537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:17.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.210+0000 7fe7537fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:17.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.210+0000 7fe7537fe640 10 monclient: dump: 2024-03-25T19:41:17.211 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:17.211 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:17.211 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:17.211 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:17.211 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:17.211 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:17.212 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:17.212 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:17.212 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:17.212 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:17.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.210+0000 7fe762cfb640 5 monclient: authenticate success, global_id 4483 2024-03-25T19:41:17.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.210+0000 7fe762cfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:17.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.219+0000 7fe7537fe640 1 -- 172.21.15.44:0/180770881 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe754001b70 con 0x7fe75c1430c0 2024-03-25T19:41:17.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.219+0000 7fe7537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:17.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.219+0000 7fe7537fe640 1 -- 172.21.15.44:0/180770881 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fe754001f80 con 0x7fe75c1430c0 2024-03-25T19:41:17.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.219+0000 7fe7537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:17.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.219+0000 7fe7537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:17.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.219+0000 7fe7537fe640 10 monclient: dump: 2024-03-25T19:41:17.219 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:17.219 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:17.219 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:17.219 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:17.219 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:17.219 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:17.219 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:17.219 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:17.220 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:17.220 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:17.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.219+0000 7fe762cfb640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:17.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.219+0000 7fe762cfb640 4 set_mon_vals no callback set 2024-03-25T19:41:17.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.219+0000 7fe762cfb640 10 monclient: shutdown 2024-03-25T19:41:17.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.220+0000 7fe762cfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:17.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.220+0000 7fe762cfb640 1 -- 172.21.15.44:0/180770881 >> v1:172.21.15.111:6789/0 conn(0x7fe75c1430c0 legacy=0x7fe75c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:17.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.220+0000 7fe762cfb640 1 -- 172.21.15.44:0/180770881 shutdown_connections 2024-03-25T19:41:17.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.220+0000 7fe762cfb640 1 -- 172.21.15.44:0/180770881 >> 172.21.15.44:0/180770881 conn(0x7fe75c0b7ea0 msgr2=0x7fe75c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:17.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.220+0000 7fe762cfb640 1 -- 172.21.15.44:0/180770881 shutdown_connections 2024-03-25T19:41:17.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.220+0000 7fe762cfb640 1 -- 172.21.15.44:0/180770881 wait complete. 2024-03-25T19:41:17.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.220+0000 7fe762cfb640 10 monclient: build_initial_monmap 2024-03-25T19:41:17.221 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.221+0000 7fe762cfb640 10 monclient: monmap: 2024-03-25T19:41:17.221 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:41:17.222 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:17.222 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:41:17.222 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:41:17.222 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:17.222 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:41:17.222 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:41:17.222 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:41:17.222 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:41:17.222 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:17.222 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.221+0000 7fe762cfb640 1 librados: starting msgr at 2024-03-25T19:41:17.222 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.221+0000 7fe762cfb640 1 librados: starting objecter 2024-03-25T19:41:17.222 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.221+0000 7fe762cfb640 1 Processor -- start 2024-03-25T19:41:17.222 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.221+0000 7fe762cfb640 1 -- start start 2024-03-25T19:41:17.222 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.221+0000 7fe762cfb640 1 librados: setting wanted keys 2024-03-25T19:41:17.223 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.221+0000 7fe762cfb640 1 librados: calling monclient init 2024-03-25T19:41:17.223 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.221+0000 7fe762cfb640 10 monclient: init 2024-03-25T19:41:17.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.222+0000 7fe762cfb640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:17.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.222+0000 7fe762cfb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:17.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.222+0000 7fe762cfb640 10 monclient(hunting): picked mon.noname-c con 0x7fe75c0a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T19:41:17.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.222+0000 7fe762cfb640 10 monclient(hunting): picked mon.noname-a con 0x7fe75c13f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:41:17.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.222+0000 7fe762cfb640 10 monclient(hunting): picked mon.noname-b con 0x7fe75c1430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:41:17.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.222+0000 7fe762cfb640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe75c136ad0 con 0x7fe75c13f710 2024-03-25T19:41:17.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.222+0000 7fe762cfb640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe75c0b7520 con 0x7fe75c1430c0 2024-03-25T19:41:17.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.222+0000 7fe762cfb640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe75c135810 con 0x7fe75c0a98f0 2024-03-25T19:41:17.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.222+0000 7fe762cfb640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:17.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.222+0000 7fe762cfb640 10 monclient(hunting): authenticate will time out at 5741.393304s 2024-03-25T19:41:17.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.232+0000 7fe761298640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe75c1430c0 0x7fe75c130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:57854/0 (socket says 172.21.15.44:57854) 2024-03-25T19:41:17.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.232+0000 7fe761298640 1 -- 172.21.15.44:0/532260439 learned_addr learned my addr 172.21.15.44:0/532260439 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:41:17.236 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.236+0000 7fe7517fa640 10 client.?.objecter ms_handle_connect 0x7fe75c1430c0 2024-03-25T19:41:17.237 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.237+0000 7fe7517fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:17.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.241+0000 7fe7517fa640 1 -- 172.21.15.44:0/532260439 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278108292 0 0) 0x7fe75c0b7520 con 0x7fe75c1430c0 2024-03-25T19:41:17.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.241+0000 7fe7517fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:17.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.241+0000 7fe7517fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:17.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.241+0000 7fe7517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:17.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.241+0000 7fe7517fa640 10 monclient(hunting): my global_id is 4486 2024-03-25T19:41:17.242 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.241+0000 7fe7517fa640 1 -- 172.21.15.44:0/532260439 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe748004eb0 con 0x7fe75c1430c0 2024-03-25T19:41:17.249 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.250+0000 7fe7517fa640 10 client.?.objecter ms_handle_connect 0x7fe75c0a98f0 2024-03-25T19:41:17.249 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.250+0000 7fe7517fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:17.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.250+0000 7fe7517fa640 1 -- 172.21.15.44:0/532260439 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4263963171 0 0) 0x7fe748004eb0 con 0x7fe75c1430c0 2024-03-25T19:41:17.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.250+0000 7fe7517fa640 1 -- 172.21.15.44:0/532260439 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe75c0b7520 con 0x7fe75c1430c0 2024-03-25T19:41:17.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.251+0000 7fe7517fa640 1 -- 172.21.15.44:0/532260439 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fe754001f40 con 0x7fe75c1430c0 2024-03-25T19:41:17.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.251+0000 7fe7517fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:17.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.251+0000 7fe7517fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:17.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.251+0000 7fe7517fa640 10 monclient(hunting): dump: 2024-03-25T19:41:17.251 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:17.251 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:17.251 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:17.251 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:17.251 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:17.251 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:17.251 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:17.252 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:17.252 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:17.252 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:17.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.251+0000 7fe7517fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:17.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.259+0000 7fe7517fa640 10 client.?.objecter ms_handle_connect 0x7fe75c13f710 2024-03-25T19:41:17.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.259+0000 7fe7517fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:17.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.262+0000 7fe7517fa640 1 -- 172.21.15.44:0/532260439 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 647957336 0 0) 0x7fe75c0b7520 con 0x7fe75c1430c0 2024-03-25T19:41:17.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.262+0000 7fe7517fa640 1 -- 172.21.15.44:0/532260439 >> v1:172.21.15.44:6790/0 conn(0x7fe75c0a98f0 legacy=0x7fe75c135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:17.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.262+0000 7fe7517fa640 1 -- 172.21.15.44:0/532260439 >> v1:172.21.15.44:6789/0 conn(0x7fe75c13f710 legacy=0x7fe75c0b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:17.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.262+0000 7fe7517fa640 10 monclient: _finish_hunting 0 2024-03-25T19:41:17.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.262+0000 7fe7517fa640 1 monclient: found mon.b 2024-03-25T19:41:17.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.262+0000 7fe7517fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:17.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.262+0000 7fe7517fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:41:17.262 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.262+0000 7fe7517fa640 1 -- 172.21.15.44:0/532260439 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75c0b6180 con 0x7fe75c1430c0 2024-03-25T19:41:17.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.262+0000 7fe7517fa640 10 monclient: _finish_auth 0 2024-03-25T19:41:17.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.262+0000 7fe7517fa640 10 monclient: _check_auth_tickets 2024-03-25T19:41:17.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.262+0000 7fe7517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:17.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.262+0000 7fe762cfb640 5 monclient: authenticate success, global_id 4486 2024-03-25T19:41:17.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.262+0000 7fe762cfb640 10 monclient: _renew_subs 2024-03-25T19:41:17.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.262+0000 7fe762cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:41:17.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.262+0000 7fe762cfb640 1 -- 172.21.15.44:0/532260439 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe75c0b7a40 con 0x7fe75c1430c0 2024-03-25T19:41:17.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.262+0000 7fe762cfb640 10 client.4486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:17.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.262+0000 7fe762cfb640 10 monclient: _renew_subs 2024-03-25T19:41:17.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.262+0000 7fe762cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:41:17.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.262+0000 7fe762cfb640 1 -- 172.21.15.44:0/532260439 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe75c131840 con 0x7fe75c1430c0 2024-03-25T19:41:17.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.262+0000 7fe762cfb640 1 librados: init done 2024-03-25T19:41:17.264 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.263+0000 7fe762cfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:17.264 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.263+0000 7fe762cfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:17.264 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.263+0000 7fe762cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:41:17.264 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.263+0000 7fe762cfb640 1 -- 172.21.15.44:0/532260439 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe75c136f60 con 0x7fe75c1430c0 2024-03-25T19:41:17.536 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.536+0000 7fe7517fa640 1 -- 172.21.15.44:0/532260439 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe754004e10 con 0x7fe75c1430c0 2024-03-25T19:41:17.536 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.536+0000 7fe7517fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:17.536 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.536+0000 7fe7517fa640 1 -- 172.21.15.44:0/532260439 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fe754004f80 con 0x7fe75c1430c0 2024-03-25T19:41:17.536 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.536+0000 7fe7517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:17.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.536+0000 7fe7517fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:17.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.536+0000 7fe7517fa640 10 monclient: dump: 2024-03-25T19:41:17.537 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:17.537 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:17.537 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:17.537 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:17.537 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:17.537 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:17.537 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:17.537 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:17.537 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:17.537 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:17.537 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.536+0000 7fe752ffd640 4 set_mon_vals no callback set 2024-03-25T19:41:17.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.537+0000 7fe7517fa640 1 -- 172.21.15.44:0/532260439 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7fe75401af50 con 0x7fe75c1430c0 2024-03-25T19:41:17.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.537+0000 7fe760a97640 10 monclient: get_auth_request con 0x7fe74806dd00 auth_method 0 2024-03-25T19:41:17.538 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.538+0000 7fe7517fa640 10 client.4486.objecter ms_handle_connect 0x7fe74806dd00 2024-03-25T19:41:17.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.541+0000 7fe7517fa640 1 -- 172.21.15.44:0/532260439 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7fe754054aa0 con 0x7fe75c1430c0 2024-03-25T19:41:17.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.541+0000 7fe7517fa640 10 client.4486.objecter ms_dispatch 0x7fe75c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:17.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.542+0000 7fe7517fa640 3 client.4486.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:17.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.542+0000 7fe7517fa640 3 client.4486.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:17.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.542+0000 7fe7517fa640 20 client.4486.objecter dump_active .. 0 homeless 2024-03-25T19:41:17.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.543+0000 7fe7517fa640 1 -- 172.21.15.44:0/532260439 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe7540505b0 con 0x7fe75c1430c0 2024-03-25T19:41:17.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.543+0000 7fe7517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:17.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.543+0000 7fe7517fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:17.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.783+0000 7fe762cfb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:17.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.783+0000 7fe762cfb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:17.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.783+0000 7fe762cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:41:17.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.783+0000 7fe762cfb640 1 -- 172.21.15.44:0/532260439 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe75c0b76b0 con 0x7fe75c1430c0 2024-03-25T19:41:17.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.816+0000 7fe7517fa640 1 -- 172.21.15.44:0/532260439 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7fe7540524e0 con 0x7fe75c1430c0 2024-03-25T19:41:17.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.816+0000 7fe7517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:17.817 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.816+0000 7fe7517fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:17.817 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:41:17.817 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:17.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.819+0000 7fe762cfb640 10 librados: watch_flush enter 2024-03-25T19:41:17.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.819+0000 7fe762cfb640 10 librados: watch_flush exit 2024-03-25T19:41:17.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.819+0000 7fe762cfb640 20 client.4486.objecter shutdown clearing up homeless session... 2024-03-25T19:41:17.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.819+0000 7fe762cfb640 10 client.4486.objecter successfully canceled tick 2024-03-25T19:41:17.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.819+0000 7fe762cfb640 1 -- 172.21.15.44:0/532260439 >> v1:172.21.15.111:6800/3883215793 conn(0x7fe74806dd00 legacy=0x7fe7480701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:17.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.819+0000 7fe762cfb640 10 monclient: shutdown 2024-03-25T19:41:17.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.819+0000 7fe762cfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:17.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.819+0000 7fe762cfb640 1 -- 172.21.15.44:0/532260439 >> v1:172.21.15.111:6789/0 conn(0x7fe75c1430c0 legacy=0x7fe75c130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:17.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.819+0000 7fe762cfb640 1 -- 172.21.15.44:0/532260439 shutdown_connections 2024-03-25T19:41:17.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.819+0000 7fe762cfb640 1 -- 172.21.15.44:0/532260439 >> 172.21.15.44:0/532260439 conn(0x7fe75c0b7ea0 msgr2=0x7fe75c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:17.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.819+0000 7fe762cfb640 1 -- 172.21.15.44:0/532260439 shutdown_connections 2024-03-25T19:41:17.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.819+0000 7fe762cfb640 1 -- 172.21.15.44:0/532260439 wait complete. 2024-03-25T19:41:17.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:17.819+0000 7fe762cfb640 1 librados: shutdown 2024-03-25T19:41:17.831 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:19.676 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:06:10.577 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:19.685 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:06:10.587 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:19.709 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:06:10.611 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:19.733 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:06:10.613 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:19.737 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:06:10.639 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:19.739 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:06:10.619 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:19.747 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:06:10.649 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:19.766 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:06:10.646 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:19.776 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:06:10.678 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:24.637 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:24.746 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.746+0000 7f5a74fa8640 10 monclient: get_monmap_and_config 2024-03-25T19:41:24.746 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.746+0000 7f5a74fa8640 10 monclient: build_initial_monmap 2024-03-25T19:41:24.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.747+0000 7f5a74fa8640 10 monclient: monmap: 2024-03-25T19:41:24.747 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:41:24.747 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:24.747 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:41:24.748135+0000 2024-03-25T19:41:24.747 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:41:24.748135+0000 2024-03-25T19:41:24.747 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:24.747 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:24.747 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:41:24.747 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:41:24.747 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:41:24.747 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:24.748 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.747+0000 7f5a74fa8640 1 Processor -- start 2024-03-25T19:41:24.748 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.747+0000 7f5a74fa8640 1 -- start start 2024-03-25T19:41:24.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.747+0000 7f5a74fa8640 10 monclient: init 2024-03-25T19:41:24.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.748+0000 7f5a74fa8640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:24.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.748+0000 7f5a74fa8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:24.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.748+0000 7f5a74fa8640 10 monclient(hunting): picked mon.noname-b con 0x7f5a700b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T19:41:24.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.748+0000 7f5a74fa8640 10 monclient(hunting): picked mon.noname-c con 0x7f5a700b1730 addr v1:172.21.15.111:6789/0 2024-03-25T19:41:24.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.748+0000 7f5a74fa8640 10 monclient(hunting): picked mon.noname-a con 0x7f5a7014be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:41:24.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.748+0000 7f5a74fa8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a700af940 con 0x7f5a7014be90 2024-03-25T19:41:24.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.748+0000 7f5a74fa8640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a700b0a70 con 0x7f5a700b1730 2024-03-25T19:41:24.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.748+0000 7f5a74fa8640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a700b0d50 con 0x7f5a700b4d60 2024-03-25T19:41:24.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.748+0000 7f5a74fa8640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:24.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.748+0000 7f5a74fa8640 10 monclient(hunting): authenticate will time out at 5748.919675s 2024-03-25T19:41:24.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.753+0000 7f5a65d9b640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f5a700b1730 0x7f5a700b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:51558/0 (socket says 172.21.15.44:51558) 2024-03-25T19:41:24.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.753+0000 7f5a65d9b640 1 -- 172.21.15.44:0/4203421098 learned_addr learned my addr 172.21.15.44:0/4203421098 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:41:24.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.769+0000 7f5a6dd9b640 1 -- 172.21.15.44:0/4203421098 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266562601 0 0) 0x7f5a700b0d50 con 0x7f5a700b4d60 2024-03-25T19:41:24.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.769+0000 7f5a6dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:41:24.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.769+0000 7f5a6dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:24.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.769+0000 7f5a6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:24.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.769+0000 7f5a6dd9b640 10 monclient(hunting): my global_id is 4478 2024-03-25T19:41:24.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.769+0000 7f5a6dd9b640 1 -- 172.21.15.44:0/4203421098 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a5c003520 con 0x7f5a700b4d60 2024-03-25T19:41:24.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.769+0000 7f5a6dd9b640 1 -- 172.21.15.44:0/4203421098 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592508253 0 0) 0x7f5a700b0a70 con 0x7f5a700b1730 2024-03-25T19:41:24.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.769+0000 7f5a6dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:41:24.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.769+0000 7f5a6dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:24.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.769+0000 7f5a6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:24.770 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.769+0000 7f5a6dd9b640 10 monclient(hunting): my global_id is 4489 2024-03-25T19:41:24.770 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.769+0000 7f5a6dd9b640 1 -- 172.21.15.44:0/4203421098 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a700b0d50 con 0x7f5a700b1730 2024-03-25T19:41:24.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.783+0000 7f5a6dd9b640 1 -- 172.21.15.44:0/4203421098 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2717657913 0 0) 0x7f5a700af940 con 0x7f5a7014be90 2024-03-25T19:41:24.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.783+0000 7f5a6dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:41:24.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.783+0000 7f5a6dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:24.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.783+0000 7f5a6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:24.784 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.783+0000 7f5a6dd9b640 10 monclient(hunting): my global_id is 4503 2024-03-25T19:41:24.784 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.783+0000 7f5a6dd9b640 1 -- 172.21.15.44:0/4203421098 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a700b0a70 con 0x7f5a7014be90 2024-03-25T19:41:24.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.786+0000 7f5a6dd9b640 1 -- 172.21.15.44:0/4203421098 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1014414619 0 0) 0x7f5a5c003520 con 0x7f5a700b4d60 2024-03-25T19:41:24.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.786+0000 7f5a6dd9b640 1 -- 172.21.15.44:0/4203421098 >> v1:172.21.15.111:6789/0 conn(0x7f5a700b1730 legacy=0x7f5a700b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:24.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.786+0000 7f5a6dd9b640 1 -- 172.21.15.44:0/4203421098 >> v1:172.21.15.44:6789/0 conn(0x7f5a7014be90 legacy=0x7f5a7014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:24.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.786+0000 7f5a6dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:41:24.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.786+0000 7f5a6dd9b640 1 monclient: found mon.noname-b 2024-03-25T19:41:24.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.786+0000 7f5a6dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:24.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.786+0000 7f5a6dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:41:24.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.786+0000 7f5a6dd9b640 1 -- 172.21.15.44:0/4203421098 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a700b1030 con 0x7f5a700b4d60 2024-03-25T19:41:24.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.786+0000 7f5a6dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:41:24.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.786+0000 7f5a6dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:41:24.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.786+0000 7f5a6dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:24.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.786+0000 7f5a6dd9b640 1 -- 172.21.15.44:0/4203421098 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f5a60001690 con 0x7f5a700b4d60 2024-03-25T19:41:24.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.786+0000 7f5a6dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:24.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.786+0000 7f5a6dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:24.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.786+0000 7f5a6dd9b640 10 monclient: dump: 2024-03-25T19:41:24.787 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:24.787 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:24.787 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:24.787 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:24.787 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:24.787 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:24.787 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:24.787 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:24.788 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:24.788 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:24.788 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.786+0000 7f5a74fa8640 5 monclient: authenticate success, global_id 4478 2024-03-25T19:41:24.788 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.786+0000 7f5a74fa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:24.798 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.798+0000 7f5a6dd9b640 1 -- 172.21.15.44:0/4203421098 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a600017f0 con 0x7f5a700b4d60 2024-03-25T19:41:24.798 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.798+0000 7f5a6dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:24.798 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.798+0000 7f5a6dd9b640 1 -- 172.21.15.44:0/4203421098 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f5a60001c00 con 0x7f5a700b4d60 2024-03-25T19:41:24.798 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.798+0000 7f5a6dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:24.798 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.798+0000 7f5a6dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:24.798 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.798+0000 7f5a6dd9b640 10 monclient: dump: 2024-03-25T19:41:24.798 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:24.798 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:24.798 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:24.798 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:24.798 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:24.798 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:24.799 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:24.799 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:24.799 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:24.799 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:24.799 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.798+0000 7f5a74fa8640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:24.799 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.798+0000 7f5a74fa8640 4 set_mon_vals no callback set 2024-03-25T19:41:24.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.799+0000 7f5a74fa8640 10 monclient: shutdown 2024-03-25T19:41:24.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.799+0000 7f5a74fa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:24.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.799+0000 7f5a74fa8640 1 -- 172.21.15.44:0/4203421098 >> v1:172.21.15.44:6790/0 conn(0x7f5a700b4d60 legacy=0x7f5a7013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:24.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.799+0000 7f5a74fa8640 1 -- 172.21.15.44:0/4203421098 shutdown_connections 2024-03-25T19:41:24.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.799+0000 7f5a74fa8640 1 -- 172.21.15.44:0/4203421098 >> 172.21.15.44:0/4203421098 conn(0x7f5a700617a0 msgr2=0x7f5a70061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:24.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.799+0000 7f5a74fa8640 1 -- 172.21.15.44:0/4203421098 shutdown_connections 2024-03-25T19:41:24.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.799+0000 7f5a74fa8640 1 -- 172.21.15.44:0/4203421098 wait complete. 2024-03-25T19:41:24.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.800+0000 7f5a74fa8640 10 monclient: build_initial_monmap 2024-03-25T19:41:24.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.800+0000 7f5a74fa8640 10 monclient: monmap: 2024-03-25T19:41:24.800 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:41:24.801 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:24.801 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:41:24.801 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:41:24.801 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:24.801 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:41:24.801 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:41:24.801 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:41:24.801 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:41:24.801 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:24.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.800+0000 7f5a74fa8640 1 librados: starting msgr at 2024-03-25T19:41:24.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.800+0000 7f5a74fa8640 1 librados: starting objecter 2024-03-25T19:41:24.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.801+0000 7f5a74fa8640 1 Processor -- start 2024-03-25T19:41:24.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.801+0000 7f5a74fa8640 1 -- start start 2024-03-25T19:41:24.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.801+0000 7f5a74fa8640 1 librados: setting wanted keys 2024-03-25T19:41:24.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.801+0000 7f5a74fa8640 1 librados: calling monclient init 2024-03-25T19:41:24.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.801+0000 7f5a74fa8640 10 monclient: init 2024-03-25T19:41:24.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.802+0000 7f5a74fa8640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:24.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.802+0000 7f5a74fa8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:24.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.802+0000 7f5a74fa8640 10 monclient(hunting): picked mon.noname-a con 0x7f5a700b1730 addr v1:172.21.15.44:6789/0 2024-03-25T19:41:24.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.802+0000 7f5a74fa8640 10 monclient(hunting): picked mon.noname-b con 0x7f5a700b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T19:41:24.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.802+0000 7f5a74fa8640 10 monclient(hunting): picked mon.noname-c con 0x7f5a7014be90 addr v1:172.21.15.44:6790/0 2024-03-25T19:41:24.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.802+0000 7f5a74fa8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a701d7980 con 0x7f5a700b1730 2024-03-25T19:41:24.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.802+0000 7f5a74fa8640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a701420a0 con 0x7f5a700b4d60 2024-03-25T19:41:24.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.802+0000 7f5a74fa8640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a701432c0 con 0x7f5a7014be90 2024-03-25T19:41:24.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.802+0000 7f5a74fa8640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:24.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.802+0000 7f5a74fa8640 10 monclient(hunting): authenticate will time out at 5748.973161s 2024-03-25T19:41:24.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.810+0000 7f5a6e59c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f5a700b1730 0x7f5a70141990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:58250/0 (socket says 172.21.15.44:58250) 2024-03-25T19:41:24.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.810+0000 7f5a6e59c640 1 -- 172.21.15.44:0/3740080083 learned_addr learned my addr 172.21.15.44:0/3740080083 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:41:24.811 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.811+0000 7f5a677fe640 10 client.?.objecter ms_handle_connect 0x7f5a700b1730 2024-03-25T19:41:24.811 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.811+0000 7f5a677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:24.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.818+0000 7f5a677fe640 1 -- 172.21.15.44:0/3740080083 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1016658422 0 0) 0x7f5a701d7980 con 0x7f5a700b1730 2024-03-25T19:41:24.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.818+0000 7f5a677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:24.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.818+0000 7f5a677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:24.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.818+0000 7f5a677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:24.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.818+0000 7f5a677fe640 10 monclient(hunting): my global_id is 4506 2024-03-25T19:41:24.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.818+0000 7f5a677fe640 1 -- 172.21.15.44:0/3740080083 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a50004eb0 con 0x7f5a700b1730 2024-03-25T19:41:24.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.821+0000 7f5a677fe640 10 client.?.objecter ms_handle_connect 0x7f5a7014be90 2024-03-25T19:41:24.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.821+0000 7f5a677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:24.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.824+0000 7f5a677fe640 10 client.?.objecter ms_handle_connect 0x7f5a700b4d60 2024-03-25T19:41:24.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.824+0000 7f5a677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:24.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.832+0000 7f5a677fe640 1 -- 172.21.15.44:0/3740080083 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1142527278 0 0) 0x7f5a50004eb0 con 0x7f5a700b1730 2024-03-25T19:41:24.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.832+0000 7f5a677fe640 1 -- 172.21.15.44:0/3740080083 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a701d7980 con 0x7f5a700b1730 2024-03-25T19:41:24.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.832+0000 7f5a677fe640 1 -- 172.21.15.44:0/3740080083 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f5a60001bc0 con 0x7f5a700b1730 2024-03-25T19:41:24.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.832+0000 7f5a677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:24.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.832+0000 7f5a677fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:24.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.832+0000 7f5a677fe640 10 monclient(hunting): dump: 2024-03-25T19:41:24.832 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:24.832 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:24.832 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:24.832 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:24.832 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:24.832 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:24.832 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:24.832 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:24.832 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:24.833 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:24.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.832+0000 7f5a677fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:24.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.836+0000 7f5a677fe640 1 -- 172.21.15.44:0/3740080083 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2627165247 0 0) 0x7f5a701420a0 con 0x7f5a700b4d60 2024-03-25T19:41:24.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.836+0000 7f5a677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:24.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.836+0000 7f5a677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:24.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.836+0000 7f5a677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:24.836 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.836+0000 7f5a677fe640 10 monclient(hunting): my global_id is 4492 2024-03-25T19:41:24.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.836+0000 7f5a677fe640 1 -- 172.21.15.44:0/3740080083 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a50004eb0 con 0x7f5a700b4d60 2024-03-25T19:41:24.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.836+0000 7f5a677fe640 1 -- 172.21.15.44:0/3740080083 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021448839 0 0) 0x7f5a701432c0 con 0x7f5a7014be90 2024-03-25T19:41:24.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.836+0000 7f5a677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:24.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.836+0000 7f5a677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:24.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.836+0000 7f5a677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:24.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.836+0000 7f5a677fe640 10 monclient(hunting): my global_id is 4481 2024-03-25T19:41:24.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.836+0000 7f5a677fe640 1 -- 172.21.15.44:0/3740080083 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a701420a0 con 0x7f5a7014be90 2024-03-25T19:41:24.843 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.842+0000 7f5a677fe640 1 -- 172.21.15.44:0/3740080083 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4030170888 0 0) 0x7f5a701d7980 con 0x7f5a700b1730 2024-03-25T19:41:24.843 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.842+0000 7f5a677fe640 1 -- 172.21.15.44:0/3740080083 >> v1:172.21.15.44:6790/0 conn(0x7f5a7014be90 legacy=0x7f5a701d7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:24.843 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.842+0000 7f5a677fe640 1 -- 172.21.15.44:0/3740080083 >> v1:172.21.15.111:6789/0 conn(0x7f5a700b4d60 legacy=0x7f5a70138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:24.843 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.842+0000 7f5a677fe640 10 monclient: _finish_hunting 0 2024-03-25T19:41:24.843 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.842+0000 7f5a677fe640 1 monclient: found mon.a 2024-03-25T19:41:24.843 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.842+0000 7f5a677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:24.843 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.842+0000 7f5a677fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:41:24.843 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.842+0000 7f5a677fe640 1 -- 172.21.15.44:0/3740080083 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a70143940 con 0x7f5a700b1730 2024-03-25T19:41:24.843 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.842+0000 7f5a677fe640 10 monclient: _finish_auth 0 2024-03-25T19:41:24.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.842+0000 7f5a677fe640 10 monclient: _check_auth_tickets 2024-03-25T19:41:24.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.842+0000 7f5a677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:24.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.842+0000 7f5a74fa8640 5 monclient: authenticate success, global_id 4506 2024-03-25T19:41:24.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.842+0000 7f5a74fa8640 10 monclient: _renew_subs 2024-03-25T19:41:24.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.842+0000 7f5a74fa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:41:24.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.842+0000 7f5a74fa8640 1 -- 172.21.15.44:0/3740080083 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a70138a70 con 0x7f5a700b1730 2024-03-25T19:41:24.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.842+0000 7f5a74fa8640 10 client.4506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:24.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.842+0000 7f5a74fa8640 10 monclient: _renew_subs 2024-03-25T19:41:24.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.842+0000 7f5a74fa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:41:24.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.842+0000 7f5a74fa8640 1 -- 172.21.15.44:0/3740080083 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a701d8700 con 0x7f5a700b1730 2024-03-25T19:41:24.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.842+0000 7f5a74fa8640 1 librados: init done 2024-03-25T19:41:24.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.843+0000 7f5a74fa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:24.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.843+0000 7f5a74fa8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:24.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.843+0000 7f5a74fa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:41:24.844 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:24.843+0000 7f5a74fa8640 1 -- 172.21.15.44:0/3740080083 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a70143130 con 0x7f5a700b1730 2024-03-25T19:41:25.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.152+0000 7f5a677fe640 1 -- 172.21.15.44:0/3740080083 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a600011f0 con 0x7f5a700b1730 2024-03-25T19:41:25.152 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.152+0000 7f5a677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:25.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.152+0000 7f5a677fe640 1 -- 172.21.15.44:0/3740080083 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f5a60003b00 con 0x7f5a700b1730 2024-03-25T19:41:25.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.152+0000 7f5a677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:25.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.152+0000 7f5a677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:25.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.152+0000 7f5a677fe640 10 monclient: dump: 2024-03-25T19:41:25.153 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:25.153 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:25.153 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:25.153 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:25.153 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:25.153 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:25.153 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:25.153 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:25.153 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:25.153 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:25.153 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.152+0000 7f5a6d59a640 4 set_mon_vals no callback set 2024-03-25T19:41:25.154 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.153+0000 7f5a677fe640 1 -- 172.21.15.44:0/3740080083 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f5a60003d20 con 0x7f5a700b1730 2024-03-25T19:41:25.154 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.154+0000 7f5a65d9b640 10 monclient: get_auth_request con 0x7f5a5006ddc0 auth_method 0 2024-03-25T19:41:25.154 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.154+0000 7f5a677fe640 10 client.4506.objecter ms_handle_connect 0x7f5a5006ddc0 2024-03-25T19:41:25.159 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.159+0000 7f5a677fe640 1 -- 172.21.15.44:0/3740080083 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f5a60054960 con 0x7f5a700b1730 2024-03-25T19:41:25.159 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.159+0000 7f5a677fe640 10 client.4506.objecter ms_dispatch 0x7f5a70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:25.159 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.159+0000 7f5a677fe640 3 client.4506.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:25.159 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.159+0000 7f5a677fe640 3 client.4506.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:25.159 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.159+0000 7f5a677fe640 20 client.4506.objecter dump_active .. 0 homeless 2024-03-25T19:41:25.159 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.159+0000 7f5a677fe640 1 -- 172.21.15.44:0/3740080083 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a600502a0 con 0x7f5a700b1730 2024-03-25T19:41:25.159 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.159+0000 7f5a677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:25.159 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.159+0000 7f5a677fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:25.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.425+0000 7f5a74fa8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:25.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.425+0000 7f5a74fa8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:25.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.425+0000 7f5a74fa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:41:25.426 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.425+0000 7f5a74fa8640 1 -- 172.21.15.44:0/3740080083 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a701401f0 con 0x7f5a700b1730 2024-03-25T19:41:25.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.458+0000 7f5a677fe640 1 -- 172.21.15.44:0/3740080083 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f5a600523a0 con 0x7f5a700b1730 2024-03-25T19:41:25.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.459+0000 7f5a677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:25.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.459+0000 7f5a677fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:25.459 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:41:25.459 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:25.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.461+0000 7f5a74fa8640 10 librados: watch_flush enter 2024-03-25T19:41:25.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.461+0000 7f5a74fa8640 10 librados: watch_flush exit 2024-03-25T19:41:25.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.461+0000 7f5a74fa8640 20 client.4506.objecter shutdown clearing up homeless session... 2024-03-25T19:41:25.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.461+0000 7f5a74fa8640 10 client.4506.objecter successfully canceled tick 2024-03-25T19:41:25.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.461+0000 7f5a74fa8640 1 -- 172.21.15.44:0/3740080083 >> v1:172.21.15.111:6800/3883215793 conn(0x7f5a5006ddc0 legacy=0x7f5a50070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:25.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.461+0000 7f5a74fa8640 10 monclient: shutdown 2024-03-25T19:41:25.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.461+0000 7f5a74fa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:25.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.461+0000 7f5a74fa8640 1 -- 172.21.15.44:0/3740080083 >> v1:172.21.15.44:6789/0 conn(0x7f5a700b1730 legacy=0x7f5a70141990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:25.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.461+0000 7f5a74fa8640 1 -- 172.21.15.44:0/3740080083 shutdown_connections 2024-03-25T19:41:25.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.461+0000 7f5a74fa8640 1 -- 172.21.15.44:0/3740080083 >> 172.21.15.44:0/3740080083 conn(0x7f5a700617a0 msgr2=0x7f5a70061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:25.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.461+0000 7f5a74fa8640 1 -- 172.21.15.44:0/3740080083 shutdown_connections 2024-03-25T19:41:25.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.461+0000 7f5a74fa8640 1 -- 172.21.15.44:0/3740080083 wait complete. 2024-03-25T19:41:25.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:25.461+0000 7f5a74fa8640 1 librados: shutdown 2024-03-25T19:41:25.471 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:32.278 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:32.395 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.395+0000 7f47977a8640 10 monclient: get_monmap_and_config 2024-03-25T19:41:32.395 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.395+0000 7f47977a8640 10 monclient: build_initial_monmap 2024-03-25T19:41:32.395 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.395+0000 7f47977a8640 10 monclient: monmap: 2024-03-25T19:41:32.395 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:41:32.395 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:32.395 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:41:32.396406+0000 2024-03-25T19:41:32.395 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:41:32.396406+0000 2024-03-25T19:41:32.395 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:32.395 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:32.395 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:41:32.395 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:41:32.396 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:41:32.396 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:32.397 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.395+0000 7f47977a8640 1 Processor -- start 2024-03-25T19:41:32.397 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.396+0000 7f47977a8640 1 -- start start 2024-03-25T19:41:32.397 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.396+0000 7f47977a8640 10 monclient: init 2024-03-25T19:41:32.397 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.396+0000 7f47977a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:32.397 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.396+0000 7f47977a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:32.397 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.396+0000 7f47977a8640 10 monclient(hunting): picked mon.noname-a con 0x7f47900a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:41:32.397 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.396+0000 7f47977a8640 10 monclient(hunting): picked mon.noname-b con 0x7f479013f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:41:32.397 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.396+0000 7f47977a8640 10 monclient(hunting): picked mon.noname-c con 0x7f47901430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:41:32.397 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.396+0000 7f47977a8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4790142520 con 0x7f47900a98f0 2024-03-25T19:41:32.398 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.396+0000 7f47977a8640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47900b28a0 con 0x7f47901430c0 2024-03-25T19:41:32.398 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.396+0000 7f47977a8640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47900b2a30 con 0x7f479013f710 2024-03-25T19:41:32.398 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.396+0000 7f47977a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:32.398 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.396+0000 7f47977a8640 10 monclient(hunting): authenticate will time out at 5756.567751s 2024-03-25T19:41:32.403 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.403+0000 7f4795d45640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f47901430c0 0x7f479014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:45086/0 (socket says 172.21.15.44:45086) 2024-03-25T19:41:32.403 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.403+0000 7f4795d45640 1 -- 172.21.15.44:0/3722493743 learned_addr learned my addr 172.21.15.44:0/3722493743 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:41:32.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.447+0000 7f4787fff640 1 -- 172.21.15.44:0/3722493743 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801003171 0 0) 0x7f4790142520 con 0x7f47900a98f0 2024-03-25T19:41:32.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.447+0000 7f4787fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:32.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.447+0000 7f4787fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:32.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.447+0000 7f4787fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:32.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.447+0000 7f4787fff640 10 monclient(hunting): my global_id is 4509 2024-03-25T19:41:32.448 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.447+0000 7f4787fff640 1 -- 172.21.15.44:0/3722493743 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f477c003520 con 0x7f47900a98f0 2024-03-25T19:41:32.529 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.530+0000 7f4787fff640 1 -- 172.21.15.44:0/3722493743 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559969347 0 0) 0x7f47900b28a0 con 0x7f47901430c0 2024-03-25T19:41:32.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.530+0000 7f4787fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:32.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.530+0000 7f4787fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:32.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.530+0000 7f4787fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:32.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.530+0000 7f4787fff640 10 monclient(hunting): my global_id is 4495 2024-03-25T19:41:32.530 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.530+0000 7f4787fff640 1 -- 172.21.15.44:0/3722493743 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4790142520 con 0x7f47901430c0 2024-03-25T19:41:32.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.540+0000 7f4787fff640 1 -- 172.21.15.44:0/3722493743 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 158274210 0 0) 0x7f4790142520 con 0x7f47901430c0 2024-03-25T19:41:32.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.540+0000 7f4787fff640 1 -- 172.21.15.44:0/3722493743 >> v1:172.21.15.44:6790/0 conn(0x7f479013f710 legacy=0x7f47901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:32.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.540+0000 7f4787fff640 1 -- 172.21.15.44:0/3722493743 >> v1:172.21.15.44:6789/0 conn(0x7f47900a98f0 legacy=0x7f47900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:32.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.540+0000 7f4787fff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:32.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.540+0000 7f4787fff640 1 monclient: found mon.noname-c 2024-03-25T19:41:32.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.540+0000 7f4787fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:32.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.540+0000 7f4787fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:41:32.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.540+0000 7f4787fff640 1 -- 172.21.15.44:0/3722493743 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47900b44c0 con 0x7f47901430c0 2024-03-25T19:41:32.540 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.540+0000 7f4787fff640 10 monclient: _finish_auth 0 2024-03-25T19:41:32.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.540+0000 7f4787fff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:32.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.540+0000 7f4787fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:32.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.540+0000 7f4787fff640 1 -- 172.21.15.44:0/3722493743 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f47880019f0 con 0x7f47901430c0 2024-03-25T19:41:32.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.540+0000 7f47977a8640 5 monclient: authenticate success, global_id 4495 2024-03-25T19:41:32.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.540+0000 7f47977a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:32.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.540+0000 7f4787fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:32.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.540+0000 7f4787fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:32.541 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.540+0000 7f4787fff640 10 monclient: dump: 2024-03-25T19:41:32.541 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:32.541 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:32.541 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:32.541 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:32.541 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:32.541 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:32.541 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:32.542 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:32.542 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:32.542 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:32.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.540+0000 7f47977a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:32.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.542+0000 7f4787fff640 1 -- 172.21.15.44:0/3722493743 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4788001b50 con 0x7f47901430c0 2024-03-25T19:41:32.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.542+0000 7f4787fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:32.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.542+0000 7f4787fff640 1 -- 172.21.15.44:0/3722493743 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f4788001f20 con 0x7f47901430c0 2024-03-25T19:41:32.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.543+0000 7f4787fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:32.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.543+0000 7f4787fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:32.543 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.543+0000 7f4787fff640 10 monclient: dump: 2024-03-25T19:41:32.543 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:32.543 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:32.543 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:32.543 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:32.543 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:32.543 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:32.543 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:32.543 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:32.543 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:32.544 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:32.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.543+0000 7f47977a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:32.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.543+0000 7f47977a8640 4 set_mon_vals no callback set 2024-03-25T19:41:32.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.543+0000 7f47977a8640 10 monclient: shutdown 2024-03-25T19:41:32.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.543+0000 7f47977a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:32.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.543+0000 7f47977a8640 1 -- 172.21.15.44:0/3722493743 >> v1:172.21.15.111:6789/0 conn(0x7f47901430c0 legacy=0x7f479014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:32.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.543+0000 7f47977a8640 1 -- 172.21.15.44:0/3722493743 shutdown_connections 2024-03-25T19:41:32.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.543+0000 7f47977a8640 1 -- 172.21.15.44:0/3722493743 >> 172.21.15.44:0/3722493743 conn(0x7f47900b7ea0 msgr2=0x7f47900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:32.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.544+0000 7f47977a8640 1 -- 172.21.15.44:0/3722493743 shutdown_connections 2024-03-25T19:41:32.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.544+0000 7f47977a8640 1 -- 172.21.15.44:0/3722493743 wait complete. 2024-03-25T19:41:32.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.544+0000 7f47977a8640 10 monclient: build_initial_monmap 2024-03-25T19:41:32.546 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.544+0000 7f47977a8640 10 monclient: monmap: 2024-03-25T19:41:32.546 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:41:32.546 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:32.546 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:41:32.546 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:41:32.546 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:32.546 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:41:32.547 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:41:32.547 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:41:32.547 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:41:32.547 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:32.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.544+0000 7f47977a8640 1 librados: starting msgr at 2024-03-25T19:41:32.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.544+0000 7f47977a8640 1 librados: starting objecter 2024-03-25T19:41:32.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.545+0000 7f47977a8640 1 Processor -- start 2024-03-25T19:41:32.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.545+0000 7f47977a8640 1 -- start start 2024-03-25T19:41:32.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.545+0000 7f47977a8640 1 librados: setting wanted keys 2024-03-25T19:41:32.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.545+0000 7f47977a8640 1 librados: calling monclient init 2024-03-25T19:41:32.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.545+0000 7f47977a8640 10 monclient: init 2024-03-25T19:41:32.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.545+0000 7f47977a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:32.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.545+0000 7f47977a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:32.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.545+0000 7f47977a8640 10 monclient(hunting): picked mon.noname-b con 0x7f47900a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:41:32.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.545+0000 7f47977a8640 10 monclient(hunting): picked mon.noname-a con 0x7f479013f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:41:32.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.545+0000 7f47977a8640 10 monclient(hunting): picked mon.noname-c con 0x7f47901430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:41:32.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.545+0000 7f47977a8640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47900ad410 con 0x7f479013f710 2024-03-25T19:41:32.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.545+0000 7f47977a8640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47901d7d80 con 0x7f47900a98f0 2024-03-25T19:41:32.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.545+0000 7f47977a8640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47901d8360 con 0x7f47901430c0 2024-03-25T19:41:32.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.545+0000 7f47977a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:32.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.545+0000 7f47977a8640 10 monclient(hunting): authenticate will time out at 5756.716595s 2024-03-25T19:41:32.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.555+0000 7f4795544640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f47900a98f0 0x7f47900b10c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:45130/0 (socket says 172.21.15.44:45130) 2024-03-25T19:41:32.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.555+0000 7f4795544640 1 -- 172.21.15.44:0/1057543338 learned_addr learned my addr 172.21.15.44:0/1057543338 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:41:32.557 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.557+0000 7f4785ffb640 10 client.?.objecter ms_handle_connect 0x7f479013f710 2024-03-25T19:41:32.557 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.557+0000 7f4785ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:32.557 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.557+0000 7f4785ffb640 10 client.?.objecter ms_handle_connect 0x7f47900a98f0 2024-03-25T19:41:32.557 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.557+0000 7f4785ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:32.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.565+0000 7f4785ffb640 1 -- 172.21.15.44:0/1057543338 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214747635 0 0) 0x7f47900ad410 con 0x7f479013f710 2024-03-25T19:41:32.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.565+0000 7f4785ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:32.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.565+0000 7f4785ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:32.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.565+0000 7f4785ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:32.565 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.565+0000 7f4785ffb640 10 monclient(hunting): my global_id is 4512 2024-03-25T19:41:32.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.565+0000 7f4785ffb640 1 -- 172.21.15.44:0/1057543338 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4774004eb0 con 0x7f479013f710 2024-03-25T19:41:32.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.576+0000 7f4785ffb640 1 -- 172.21.15.44:0/1057543338 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053132559 0 0) 0x7f47901d7d80 con 0x7f47900a98f0 2024-03-25T19:41:32.575 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.576+0000 7f4785ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:32.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.576+0000 7f4785ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:32.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.576+0000 7f4785ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:32.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.576+0000 7f4785ffb640 10 monclient(hunting): my global_id is 4498 2024-03-25T19:41:32.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.576+0000 7f4785ffb640 1 -- 172.21.15.44:0/1057543338 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47900ad410 con 0x7f47900a98f0 2024-03-25T19:41:32.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.576+0000 7f4785ffb640 10 client.?.objecter ms_handle_connect 0x7f47901430c0 2024-03-25T19:41:32.576 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.576+0000 7f4785ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:32.580 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.580+0000 7f4785ffb640 1 -- 172.21.15.44:0/1057543338 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1958347016 0 0) 0x7f4774004eb0 con 0x7f479013f710 2024-03-25T19:41:32.581 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.580+0000 7f4785ffb640 1 -- 172.21.15.44:0/1057543338 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47901d7d80 con 0x7f479013f710 2024-03-25T19:41:32.581 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.580+0000 7f4785ffb640 1 -- 172.21.15.44:0/1057543338 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f47800018c0 con 0x7f479013f710 2024-03-25T19:41:32.581 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.580+0000 7f4785ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:32.581 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.580+0000 7f4785ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:32.581 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.580+0000 7f4785ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:32.581 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:32.581 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:32.581 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:32.581 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:32.581 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:32.581 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:32.581 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:32.581 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:32.582 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:32.582 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:32.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.580+0000 7f4785ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:32.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.586+0000 7f4785ffb640 1 -- 172.21.15.44:0/1057543338 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3343292007 0 0) 0x7f47901d8360 con 0x7f47901430c0 2024-03-25T19:41:32.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.586+0000 7f4785ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:32.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.586+0000 7f4785ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:32.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.586+0000 7f4785ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:32.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.586+0000 7f4785ffb640 10 monclient(hunting): my global_id is 4487 2024-03-25T19:41:32.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.586+0000 7f4785ffb640 1 -- 172.21.15.44:0/1057543338 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4774004eb0 con 0x7f47901430c0 2024-03-25T19:41:32.591 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.591+0000 7f4785ffb640 1 -- 172.21.15.44:0/1057543338 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3506405631 0 0) 0x7f47900ad410 con 0x7f47900a98f0 2024-03-25T19:41:32.591 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.591+0000 7f4785ffb640 1 -- 172.21.15.44:0/1057543338 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47901d8360 con 0x7f47900a98f0 2024-03-25T19:41:32.591 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.591+0000 7f4785ffb640 1 -- 172.21.15.44:0/1057543338 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f478c0019d0 con 0x7f47900a98f0 2024-03-25T19:41:32.591 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.591+0000 7f4785ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:32.591 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.591+0000 7f4785ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:32.591 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.591+0000 7f4785ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:32.591 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:32.591 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:32.592 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:32.592 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:32.592 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:32.592 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:32.592 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:32.592 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:32.592 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:32.592 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:32.601 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.601+0000 7f4785ffb640 1 -- 172.21.15.44:0/1057543338 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 385232572 0 0) 0x7f47901d7d80 con 0x7f479013f710 2024-03-25T19:41:32.601 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.601+0000 7f4785ffb640 1 -- 172.21.15.44:0/1057543338 >> v1:172.21.15.44:6790/0 conn(0x7f47901430c0 legacy=0x7f47901d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:32.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.601+0000 7f4785ffb640 1 -- 172.21.15.44:0/1057543338 >> v1:172.21.15.111:6789/0 conn(0x7f47900a98f0 legacy=0x7f47900b10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:32.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.601+0000 7f4785ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:41:32.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.601+0000 7f4785ffb640 1 monclient: found mon.a 2024-03-25T19:41:32.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.601+0000 7f4785ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:32.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.601+0000 7f4785ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:41:32.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.601+0000 7f4785ffb640 1 -- 172.21.15.44:0/1057543338 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47901d84f0 con 0x7f479013f710 2024-03-25T19:41:32.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.601+0000 7f4785ffb640 10 monclient: _finish_auth 0 2024-03-25T19:41:32.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.601+0000 7f4785ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:41:32.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.601+0000 7f4785ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:32.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.601+0000 7f47977a8640 5 monclient: authenticate success, global_id 4512 2024-03-25T19:41:32.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.601+0000 7f47977a8640 10 monclient: _renew_subs 2024-03-25T19:41:32.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.601+0000 7f47977a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:41:32.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.601+0000 7f47977a8640 1 -- 172.21.15.44:0/1057543338 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f479014e560 con 0x7f479013f710 2024-03-25T19:41:32.602 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.601+0000 7f47977a8640 10 client.4512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:32.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.601+0000 7f47977a8640 10 monclient: _renew_subs 2024-03-25T19:41:32.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.601+0000 7f47977a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:41:32.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.601+0000 7f47977a8640 1 -- 172.21.15.44:0/1057543338 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47901d41e0 con 0x7f479013f710 2024-03-25T19:41:32.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.601+0000 7f47977a8640 1 librados: init done 2024-03-25T19:41:32.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.602+0000 7f47977a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:32.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.602+0000 7f47977a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:32.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.602+0000 7f47977a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:41:32.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.602+0000 7f47977a8640 1 -- 172.21.15.44:0/1057543338 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47900b3ef0 con 0x7f479013f710 2024-03-25T19:41:32.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.908+0000 7f4785ffb640 1 -- 172.21.15.44:0/1057543338 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4780001a20 con 0x7f479013f710 2024-03-25T19:41:32.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.909+0000 7f4785ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:32.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.909+0000 7f4785ffb640 1 -- 172.21.15.44:0/1057543338 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f4780003850 con 0x7f479013f710 2024-03-25T19:41:32.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.909+0000 7f4785ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:32.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.909+0000 7f4785ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:32.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.909+0000 7f4785ffb640 10 monclient: dump: 2024-03-25T19:41:32.909 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:32.909 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:32.909 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:32.909 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:32.910 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:32.910 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:32.910 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:32.910 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:32.910 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:32.910 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:32.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.909+0000 7f47877fe640 4 set_mon_vals no callback set 2024-03-25T19:41:32.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.909+0000 7f4785ffb640 1 -- 172.21.15.44:0/1057543338 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f47800198e0 con 0x7f479013f710 2024-03-25T19:41:32.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.910+0000 7f4795544640 10 monclient: get_auth_request con 0x7f477406df70 auth_method 0 2024-03-25T19:41:32.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.910+0000 7f4785ffb640 10 client.4512.objecter ms_handle_connect 0x7f477406df70 2024-03-25T19:41:32.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.917+0000 7f4785ffb640 1 -- 172.21.15.44:0/1057543338 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f4780053630 con 0x7f479013f710 2024-03-25T19:41:32.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.917+0000 7f4785ffb640 10 client.4512.objecter ms_dispatch 0x7f4790000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:32.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.917+0000 7f4785ffb640 3 client.4512.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:32.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.917+0000 7f4785ffb640 3 client.4512.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:32.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.917+0000 7f4785ffb640 20 client.4512.objecter dump_active .. 0 homeless 2024-03-25T19:41:32.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.917+0000 7f4785ffb640 1 -- 172.21.15.44:0/1057543338 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f478004ef70 con 0x7f479013f710 2024-03-25T19:41:32.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.917+0000 7f4785ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:32.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:32.917+0000 7f4785ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:33.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:33.168+0000 7f47977a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:33.169 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:33.168+0000 7f47977a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:33.169 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:33.168+0000 7f47977a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:41:33.169 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:33.168+0000 7f47977a8640 1 -- 172.21.15.44:0/1057543338 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47900627b0 con 0x7f479013f710 2024-03-25T19:41:33.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:33.206+0000 7f4785ffb640 1 -- 172.21.15.44:0/1057543338 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f4780051070 con 0x7f479013f710 2024-03-25T19:41:33.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:33.206+0000 7f4785ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:33.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:33.206+0000 7f4785ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:33.207 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:41:33.207 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:33.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:33.208+0000 7f47977a8640 10 librados: watch_flush enter 2024-03-25T19:41:33.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:33.208+0000 7f47977a8640 10 librados: watch_flush exit 2024-03-25T19:41:33.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:33.208+0000 7f47977a8640 20 client.4512.objecter shutdown clearing up homeless session... 2024-03-25T19:41:33.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:33.208+0000 7f47977a8640 10 client.4512.objecter successfully canceled tick 2024-03-25T19:41:33.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:33.209+0000 7f47977a8640 1 -- 172.21.15.44:0/1057543338 >> v1:172.21.15.111:6800/3883215793 conn(0x7f477406df70 legacy=0x7f4774070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:33.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:33.209+0000 7f47977a8640 10 monclient: shutdown 2024-03-25T19:41:33.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:33.209+0000 7f47977a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:33.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:33.209+0000 7f47977a8640 1 -- 172.21.15.44:0/1057543338 >> v1:172.21.15.44:6789/0 conn(0x7f479013f710 legacy=0x7f47900acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:33.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:33.209+0000 7f47977a8640 1 -- 172.21.15.44:0/1057543338 shutdown_connections 2024-03-25T19:41:33.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:33.209+0000 7f47977a8640 1 -- 172.21.15.44:0/1057543338 >> 172.21.15.44:0/1057543338 conn(0x7f47900b7ea0 msgr2=0x7f4790074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:33.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:33.209+0000 7f47977a8640 1 -- 172.21.15.44:0/1057543338 shutdown_connections 2024-03-25T19:41:33.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:33.209+0000 7f47977a8640 1 -- 172.21.15.44:0/1057543338 wait complete. 2024-03-25T19:41:33.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:33.209+0000 7f47977a8640 1 librados: shutdown 2024-03-25T19:41:33.220 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:39.677 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:06:30.579 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:39.726 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:06:30.627 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:39.763 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:06:30.665 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:39.789 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:06:30.669 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:39.795 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:06:30.675 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:39.795 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:06:30.697 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:39.800 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:06:30.702 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:39.827 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:06:30.729 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:39.835 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:06:30.715 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:40.027 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:40.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.144+0000 7f727d9ed640 10 monclient: get_monmap_and_config 2024-03-25T19:41:40.143 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.144+0000 7f727d9ed640 10 monclient: build_initial_monmap 2024-03-25T19:41:40.143 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.144+0000 7f727d9ed640 10 monclient: monmap: 2024-03-25T19:41:40.143 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:41:40.143 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:40.143 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:41:40.144243+0000 2024-03-25T19:41:40.143 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:41:40.144243+0000 2024-03-25T19:41:40.143 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:40.143 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:40.143 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:41:40.143 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:41:40.143 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:41:40.143 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:40.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.144+0000 7f727d9ed640 1 Processor -- start 2024-03-25T19:41:40.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.144+0000 7f727d9ed640 1 -- start start 2024-03-25T19:41:40.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.144+0000 7f727d9ed640 10 monclient: init 2024-03-25T19:41:40.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.145+0000 7f727d9ed640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:40.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.145+0000 7f727d9ed640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:40.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.145+0000 7f727d9ed640 10 monclient(hunting): picked mon.noname-c con 0x7f7278061830 addr v1:172.21.15.111:6789/0 2024-03-25T19:41:40.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.145+0000 7f727d9ed640 10 monclient(hunting): picked mon.noname-b con 0x7f72780ae840 addr v1:172.21.15.44:6790/0 2024-03-25T19:41:40.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.145+0000 7f727d9ed640 10 monclient(hunting): picked mon.noname-a con 0x7f727813fbd0 addr v1:172.21.15.44:6789/0 2024-03-25T19:41:40.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.145+0000 7f727d9ed640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72780ae1c0 con 0x7f727813fbd0 2024-03-25T19:41:40.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.145+0000 7f727d9ed640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f727806f7d0 con 0x7f7278061830 2024-03-25T19:41:40.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.145+0000 7f727d9ed640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f727806fa40 con 0x7f72780ae840 2024-03-25T19:41:40.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.145+0000 7f727d9ed640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:40.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.145+0000 7f727d9ed640 10 monclient(hunting): authenticate will time out at 5764.315602s 2024-03-25T19:41:40.154 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.156+0000 7f7276ffd640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f7278061830 0x7f727813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:60992/0 (socket says 172.21.15.44:60992) 2024-03-25T19:41:40.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.156+0000 7f7276ffd640 1 -- 172.21.15.44:0/2380147187 learned_addr learned my addr 172.21.15.44:0/2380147187 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:41:40.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.169+0000 7f7275ffb640 1 -- 172.21.15.44:0/2380147187 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710462794 0 0) 0x7f727806f7d0 con 0x7f7278061830 2024-03-25T19:41:40.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.169+0000 7f7275ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:40.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.169+0000 7f7275ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:40.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.169+0000 7f7275ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:40.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.169+0000 7f7275ffb640 10 monclient(hunting): my global_id is 4501 2024-03-25T19:41:40.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.169+0000 7f7275ffb640 1 -- 172.21.15.44:0/2380147187 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7260003520 con 0x7f7278061830 2024-03-25T19:41:40.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.172+0000 7f7275ffb640 1 -- 172.21.15.44:0/2380147187 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989497928 0 0) 0x7f72780ae1c0 con 0x7f727813fbd0 2024-03-25T19:41:40.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.172+0000 7f7275ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:40.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.172+0000 7f7275ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:40.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.172+0000 7f7275ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:40.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.172+0000 7f7275ffb640 10 monclient(hunting): my global_id is 4515 2024-03-25T19:41:40.171 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.172+0000 7f7275ffb640 1 -- 172.21.15.44:0/2380147187 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f727806f7d0 con 0x7f727813fbd0 2024-03-25T19:41:40.184 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.185+0000 7f7275ffb640 1 -- 172.21.15.44:0/2380147187 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1368405997 0 0) 0x7f727806f7d0 con 0x7f727813fbd0 2024-03-25T19:41:40.184 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.185+0000 7f7275ffb640 1 -- 172.21.15.44:0/2380147187 >> v1:172.21.15.44:6790/0 conn(0x7f72780ae840 legacy=0x7f7278145810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:40.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.185+0000 7f7275ffb640 1 -- 172.21.15.44:0/2380147187 >> v1:172.21.15.111:6789/0 conn(0x7f7278061830 legacy=0x7f727813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:40.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.185+0000 7f7275ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:41:40.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.185+0000 7f7275ffb640 1 monclient: found mon.noname-a 2024-03-25T19:41:40.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.185+0000 7f7275ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:40.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.185+0000 7f7275ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:41:40.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.185+0000 7f7275ffb640 1 -- 172.21.15.44:0/2380147187 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7278073040 con 0x7f727813fbd0 2024-03-25T19:41:40.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.185+0000 7f7275ffb640 10 monclient: _finish_auth 0 2024-03-25T19:41:40.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.185+0000 7f7275ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:41:40.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.185+0000 7f7275ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:40.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.185+0000 7f7275ffb640 1 -- 172.21.15.44:0/2380147187 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f72700019f0 con 0x7f727813fbd0 2024-03-25T19:41:40.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.185+0000 7f7275ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:40.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.185+0000 7f7275ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:40.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.185+0000 7f7275ffb640 10 monclient: dump: 2024-03-25T19:41:40.185 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:40.186 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:40.186 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:40.186 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:40.186 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:40.186 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:40.186 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:40.186 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:40.186 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:40.186 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:40.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.185+0000 7f727d9ed640 5 monclient: authenticate success, global_id 4515 2024-03-25T19:41:40.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.185+0000 7f727d9ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:40.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.207+0000 7f7275ffb640 1 -- 172.21.15.44:0/2380147187 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7270001b50 con 0x7f727813fbd0 2024-03-25T19:41:40.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.207+0000 7f7275ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:40.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.207+0000 7f7275ffb640 1 -- 172.21.15.44:0/2380147187 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f7270001f20 con 0x7f727813fbd0 2024-03-25T19:41:40.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.207+0000 7f7275ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:40.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.207+0000 7f7275ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:40.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.207+0000 7f7275ffb640 10 monclient: dump: 2024-03-25T19:41:40.207 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:40.207 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:40.207 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:40.207 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:40.207 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:40.207 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:40.207 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:40.207 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:40.207 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:40.207 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:40.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.207+0000 7f727d9ed640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:40.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.207+0000 7f727d9ed640 4 set_mon_vals no callback set 2024-03-25T19:41:40.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.208+0000 7f727d9ed640 10 monclient: shutdown 2024-03-25T19:41:40.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.208+0000 7f727d9ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:40.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.208+0000 7f727d9ed640 1 -- 172.21.15.44:0/2380147187 >> v1:172.21.15.44:6789/0 conn(0x7f727813fbd0 legacy=0x7f7278145f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:40.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.208+0000 7f727d9ed640 1 -- 172.21.15.44:0/2380147187 shutdown_connections 2024-03-25T19:41:40.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.208+0000 7f727d9ed640 1 -- 172.21.15.44:0/2380147187 >> 172.21.15.44:0/2380147187 conn(0x7f72780b0ef0 msgr2=0x7f72780b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:40.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.208+0000 7f727d9ed640 1 -- 172.21.15.44:0/2380147187 shutdown_connections 2024-03-25T19:41:40.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.208+0000 7f727d9ed640 1 -- 172.21.15.44:0/2380147187 wait complete. 2024-03-25T19:41:40.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.209+0000 7f727d9ed640 10 monclient: build_initial_monmap 2024-03-25T19:41:40.210 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.209+0000 7f727d9ed640 10 monclient: monmap: 2024-03-25T19:41:40.210 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:41:40.210 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:40.210 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:41:40.211 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:41:40.211 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:40.211 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:41:40.211 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:41:40.211 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:41:40.211 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:41:40.211 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:40.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.209+0000 7f727d9ed640 1 librados: starting msgr at 2024-03-25T19:41:40.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.209+0000 7f727d9ed640 1 librados: starting objecter 2024-03-25T19:41:40.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.210+0000 7f727d9ed640 1 Processor -- start 2024-03-25T19:41:40.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.210+0000 7f727d9ed640 1 -- start start 2024-03-25T19:41:40.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.210+0000 7f727d9ed640 1 librados: setting wanted keys 2024-03-25T19:41:40.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.210+0000 7f727d9ed640 1 librados: calling monclient init 2024-03-25T19:41:40.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.210+0000 7f727d9ed640 10 monclient: init 2024-03-25T19:41:40.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.210+0000 7f727d9ed640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:40.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.210+0000 7f727d9ed640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:40.211 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.210+0000 7f727d9ed640 10 monclient(hunting): picked mon.noname-c con 0x7f7278061830 addr v1:172.21.15.44:6790/0 2024-03-25T19:41:40.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.210+0000 7f727d9ed640 10 monclient(hunting): picked mon.noname-a con 0x7f72780ae840 addr v1:172.21.15.44:6789/0 2024-03-25T19:41:40.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.210+0000 7f727d9ed640 10 monclient(hunting): picked mon.noname-b con 0x7f727813fbd0 addr v1:172.21.15.111:6789/0 2024-03-25T19:41:40.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.210+0000 7f727d9ed640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72780b3a90 con 0x7f72780ae840 2024-03-25T19:41:40.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.210+0000 7f727d9ed640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72780b3720 con 0x7f727813fbd0 2024-03-25T19:41:40.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.210+0000 7f727d9ed640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72781421d0 con 0x7f7278061830 2024-03-25T19:41:40.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.210+0000 7f727d9ed640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:40.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.210+0000 7f727d9ed640 10 monclient(hunting): authenticate will time out at 5764.380725s 2024-03-25T19:41:40.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.216+0000 7f72777fe640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f727813fbd0 0x7f72781e2b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:32810/0 (socket says 172.21.15.44:32810) 2024-03-25T19:41:40.215 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.216+0000 7f72777fe640 1 -- 172.21.15.44:0/1751272378 learned_addr learned my addr 172.21.15.44:0/1751272378 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:41:40.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.217+0000 7f725ffff640 10 client.?.objecter ms_handle_connect 0x7f727813fbd0 2024-03-25T19:41:40.216 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.217+0000 7f725ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:40.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.219+0000 7f725ffff640 10 client.?.objecter ms_handle_connect 0x7f72780ae840 2024-03-25T19:41:40.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.219+0000 7f725ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:40.225 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.226+0000 7f725ffff640 1 -- 172.21.15.44:0/1751272378 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711372364 0 0) 0x7f72780b3720 con 0x7f727813fbd0 2024-03-25T19:41:40.225 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.226+0000 7f725ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:40.225 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.226+0000 7f725ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:40.225 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.226+0000 7f725ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:40.225 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.226+0000 7f725ffff640 10 monclient(hunting): my global_id is 4504 2024-03-25T19:41:40.225 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.226+0000 7f725ffff640 1 -- 172.21.15.44:0/1751272378 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7264004eb0 con 0x7f727813fbd0 2024-03-25T19:41:40.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.230+0000 7f725ffff640 10 client.?.objecter ms_handle_connect 0x7f7278061830 2024-03-25T19:41:40.229 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.230+0000 7f725ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:40.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.232+0000 7f725ffff640 1 -- 172.21.15.44:0/1751272378 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1730804561 0 0) 0x7f72780b3a90 con 0x7f72780ae840 2024-03-25T19:41:40.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.232+0000 7f725ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:40.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.232+0000 7f725ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:40.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.232+0000 7f725ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:40.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.232+0000 7f725ffff640 10 monclient(hunting): my global_id is 4518 2024-03-25T19:41:40.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.232+0000 7f725ffff640 1 -- 172.21.15.44:0/1751272378 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72780b3720 con 0x7f72780ae840 2024-03-25T19:41:40.237 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.238+0000 7f725ffff640 1 -- 172.21.15.44:0/1751272378 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 34877912 0 0) 0x7f7264004eb0 con 0x7f727813fbd0 2024-03-25T19:41:40.237 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.238+0000 7f725ffff640 1 -- 172.21.15.44:0/1751272378 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72780b3a90 con 0x7f727813fbd0 2024-03-25T19:41:40.237 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.238+0000 7f725ffff640 1 -- 172.21.15.44:0/1751272378 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f72700018b0 con 0x7f727813fbd0 2024-03-25T19:41:40.237 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.238+0000 7f725ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:40.237 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.238+0000 7f725ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:40.237 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.238+0000 7f725ffff640 10 monclient(hunting): dump: 2024-03-25T19:41:40.238 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:40.238 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:40.238 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:40.238 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:40.238 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:40.238 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:40.238 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:40.238 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:40.238 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:40.238 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:40.238 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.238+0000 7f725ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:40.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.245+0000 7f725ffff640 1 -- 172.21.15.44:0/1751272378 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2193405114 0 0) 0x7f72780b3a90 con 0x7f727813fbd0 2024-03-25T19:41:40.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.245+0000 7f725ffff640 1 -- 172.21.15.44:0/1751272378 >> v1:172.21.15.44:6790/0 conn(0x7f7278061830 legacy=0x7f7278131670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:40.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.245+0000 7f725ffff640 1 -- 172.21.15.44:0/1751272378 >> v1:172.21.15.44:6789/0 conn(0x7f72780ae840 legacy=0x7f72781d7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:40.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.245+0000 7f725ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:40.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.245+0000 7f725ffff640 1 monclient: found mon.b 2024-03-25T19:41:40.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.245+0000 7f725ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:40.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.245+0000 7f725ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:41:40.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.245+0000 7f725ffff640 1 -- 172.21.15.44:0/1751272378 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7278131d80 con 0x7f727813fbd0 2024-03-25T19:41:40.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.245+0000 7f725ffff640 10 monclient: _finish_auth 0 2024-03-25T19:41:40.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.245+0000 7f725ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:40.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.245+0000 7f725ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:40.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.245+0000 7f727d9ed640 5 monclient: authenticate success, global_id 4504 2024-03-25T19:41:40.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.245+0000 7f727d9ed640 10 monclient: _renew_subs 2024-03-25T19:41:40.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.245+0000 7f727d9ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:41:40.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.245+0000 7f727d9ed640 1 -- 172.21.15.44:0/1751272378 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7278132030 con 0x7f727813fbd0 2024-03-25T19:41:40.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.245+0000 7f727d9ed640 10 client.4504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:40.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.245+0000 7f727d9ed640 10 monclient: _renew_subs 2024-03-25T19:41:40.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.245+0000 7f727d9ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:41:40.245 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.245+0000 7f727d9ed640 1 -- 172.21.15.44:0/1751272378 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7278142e20 con 0x7f727813fbd0 2024-03-25T19:41:40.246 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.245+0000 7f727d9ed640 1 librados: init done 2024-03-25T19:41:40.246 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.246+0000 7f727d9ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:40.246 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.246+0000 7f727d9ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:40.246 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.246+0000 7f727d9ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:41:40.246 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.246+0000 7f727d9ed640 1 -- 172.21.15.44:0/1751272378 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f727813a420 con 0x7f727813fbd0 2024-03-25T19:41:40.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.516+0000 7f725ffff640 1 -- 172.21.15.44:0/1751272378 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7270003c20 con 0x7f727813fbd0 2024-03-25T19:41:40.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.516+0000 7f725ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:40.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.516+0000 7f725ffff640 1 -- 172.21.15.44:0/1751272378 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f7270004030 con 0x7f727813fbd0 2024-03-25T19:41:40.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.516+0000 7f725ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:40.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.516+0000 7f725ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:40.516 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.516+0000 7f725ffff640 10 monclient: dump: 2024-03-25T19:41:40.517 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:40.517 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:40.517 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:40.517 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:40.517 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:40.517 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:40.517 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:40.517 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:40.517 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:40.517 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:40.517 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.516+0000 7f72757fa640 4 set_mon_vals no callback set 2024-03-25T19:41:40.517 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.516+0000 7f725ffff640 1 -- 172.21.15.44:0/1751272378 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f7270004250 con 0x7f727813fbd0 2024-03-25T19:41:40.518 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.517+0000 7f7276ffd640 10 monclient: get_auth_request con 0x7f726406dda0 auth_method 0 2024-03-25T19:41:40.518 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.517+0000 7f725ffff640 10 client.4504.objecter ms_handle_connect 0x7f726406dda0 2024-03-25T19:41:40.522 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.522+0000 7f725ffff640 1 -- 172.21.15.44:0/1751272378 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f7270054c00 con 0x7f727813fbd0 2024-03-25T19:41:40.522 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.522+0000 7f725ffff640 10 client.4504.objecter ms_dispatch 0x7f7278000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:40.522 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.522+0000 7f725ffff640 3 client.4504.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:40.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.522+0000 7f725ffff640 3 client.4504.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:40.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.522+0000 7f725ffff640 20 client.4504.objecter dump_active .. 0 homeless 2024-03-25T19:41:40.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.523+0000 7f725ffff640 1 -- 172.21.15.44:0/1751272378 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7270050710 con 0x7f727813fbd0 2024-03-25T19:41:40.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.523+0000 7f725ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:40.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.523+0000 7f725ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:40.764 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.764+0000 7f727d9ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:40.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.764+0000 7f727d9ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:40.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.764+0000 7f727d9ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:41:40.765 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.764+0000 7f727d9ed640 1 -- 172.21.15.44:0/1751272378 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72780627b0 con 0x7f727813fbd0 2024-03-25T19:41:40.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.808+0000 7f725ffff640 1 -- 172.21.15.44:0/1751272378 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f7270052640 con 0x7f727813fbd0 2024-03-25T19:41:40.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.808+0000 7f725ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:40.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.808+0000 7f725ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:40.809 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:41:40.809 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:40.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.811+0000 7f727d9ed640 10 librados: watch_flush enter 2024-03-25T19:41:40.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.811+0000 7f727d9ed640 10 librados: watch_flush exit 2024-03-25T19:41:40.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.811+0000 7f727d9ed640 20 client.4504.objecter shutdown clearing up homeless session... 2024-03-25T19:41:40.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.811+0000 7f727d9ed640 10 client.4504.objecter successfully canceled tick 2024-03-25T19:41:40.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.811+0000 7f727d9ed640 1 -- 172.21.15.44:0/1751272378 >> v1:172.21.15.111:6800/3883215793 conn(0x7f726406dda0 legacy=0x7f7264070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:40.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.811+0000 7f727d9ed640 10 monclient: shutdown 2024-03-25T19:41:40.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.811+0000 7f727d9ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:40.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.811+0000 7f727d9ed640 1 -- 172.21.15.44:0/1751272378 >> v1:172.21.15.111:6789/0 conn(0x7f727813fbd0 legacy=0x7f72781e2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:40.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.811+0000 7f727d9ed640 1 -- 172.21.15.44:0/1751272378 shutdown_connections 2024-03-25T19:41:40.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.811+0000 7f727d9ed640 1 -- 172.21.15.44:0/1751272378 >> 172.21.15.44:0/1751272378 conn(0x7f72780b0ef0 msgr2=0x7f72780aecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:40.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.811+0000 7f727d9ed640 1 -- 172.21.15.44:0/1751272378 shutdown_connections 2024-03-25T19:41:40.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.811+0000 7f727d9ed640 1 -- 172.21.15.44:0/1751272378 wait complete. 2024-03-25T19:41:40.812 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:40.811+0000 7f727d9ed640 1 librados: shutdown 2024-03-25T19:41:40.823 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:47.630 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:47.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.751+0000 7ffa2d9bc640 10 monclient: get_monmap_and_config 2024-03-25T19:41:47.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.751+0000 7ffa2d9bc640 10 monclient: build_initial_monmap 2024-03-25T19:41:47.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.751+0000 7ffa2d9bc640 10 monclient: monmap: 2024-03-25T19:41:47.751 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:41:47.751 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:47.751 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:41:47.752223+0000 2024-03-25T19:41:47.751 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:41:47.752223+0000 2024-03-25T19:41:47.751 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:47.752 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:47.752 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:41:47.752 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:41:47.752 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:41:47.752 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:47.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.751+0000 7ffa2d9bc640 1 Processor -- start 2024-03-25T19:41:47.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.751+0000 7ffa2d9bc640 1 -- start start 2024-03-25T19:41:47.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.751+0000 7ffa2d9bc640 10 monclient: init 2024-03-25T19:41:47.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.752+0000 7ffa2d9bc640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:47.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.752+0000 7ffa2d9bc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:47.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.752+0000 7ffa2d9bc640 10 monclient(hunting): picked mon.noname-a con 0x7ffa280a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:41:47.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.752+0000 7ffa2d9bc640 10 monclient(hunting): picked mon.noname-b con 0x7ffa2813f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:41:47.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.752+0000 7ffa2d9bc640 10 monclient(hunting): picked mon.noname-c con 0x7ffa281430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:41:47.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.752+0000 7ffa2d9bc640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa28142520 con 0x7ffa280a98f0 2024-03-25T19:41:47.754 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.752+0000 7ffa2d9bc640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa280b28a0 con 0x7ffa281430c0 2024-03-25T19:41:47.754 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.752+0000 7ffa2d9bc640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa280b2a30 con 0x7ffa2813f710 2024-03-25T19:41:47.754 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.752+0000 7ffa2d9bc640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:47.754 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.752+0000 7ffa2d9bc640 10 monclient(hunting): authenticate will time out at 5771.923420s 2024-03-25T19:41:47.770 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.770+0000 7ffa277fe640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ffa281430c0 0x7ffa2814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:37226/0 (socket says 172.21.15.44:37226) 2024-03-25T19:41:47.770 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.771+0000 7ffa277fe640 1 -- 172.21.15.44:0/2416564882 learned_addr learned my addr 172.21.15.44:0/2416564882 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:41:47.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.786+0000 7ffa267fc640 1 -- 172.21.15.44:0/2416564882 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048439073 0 0) 0x7ffa280b28a0 con 0x7ffa281430c0 2024-03-25T19:41:47.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.786+0000 7ffa267fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:47.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.786+0000 7ffa267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:47.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.786+0000 7ffa267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:47.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.786+0000 7ffa267fc640 10 monclient(hunting): my global_id is 4507 2024-03-25T19:41:47.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.786+0000 7ffa267fc640 1 -- 172.21.15.44:0/2416564882 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa14003520 con 0x7ffa281430c0 2024-03-25T19:41:47.788 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.788+0000 7ffa267fc640 1 -- 172.21.15.44:0/2416564882 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216988500 0 0) 0x7ffa280b2a30 con 0x7ffa2813f710 2024-03-25T19:41:47.788 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.788+0000 7ffa267fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:47.788 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.788+0000 7ffa267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:47.788 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.788+0000 7ffa267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:47.788 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.788+0000 7ffa267fc640 10 monclient(hunting): my global_id is 4496 2024-03-25T19:41:47.788 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.788+0000 7ffa267fc640 1 -- 172.21.15.44:0/2416564882 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa280b28a0 con 0x7ffa2813f710 2024-03-25T19:41:47.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.795+0000 7ffa267fc640 1 -- 172.21.15.44:0/2416564882 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1450220252 0 0) 0x7ffa28142520 con 0x7ffa280a98f0 2024-03-25T19:41:47.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.795+0000 7ffa267fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:47.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.795+0000 7ffa267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:47.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.795+0000 7ffa267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:47.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.795+0000 7ffa267fc640 10 monclient(hunting): my global_id is 4521 2024-03-25T19:41:47.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.795+0000 7ffa267fc640 1 -- 172.21.15.44:0/2416564882 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa280b2a30 con 0x7ffa280a98f0 2024-03-25T19:41:47.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.796+0000 7ffa267fc640 1 -- 172.21.15.44:0/2416564882 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 257124050 0 0) 0x7ffa14003520 con 0x7ffa281430c0 2024-03-25T19:41:47.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.796+0000 7ffa267fc640 1 -- 172.21.15.44:0/2416564882 >> v1:172.21.15.44:6790/0 conn(0x7ffa2813f710 legacy=0x7ffa281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:47.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.796+0000 7ffa267fc640 1 -- 172.21.15.44:0/2416564882 >> v1:172.21.15.44:6789/0 conn(0x7ffa280a98f0 legacy=0x7ffa280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:47.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.796+0000 7ffa267fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:47.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.796+0000 7ffa267fc640 1 monclient: found mon.noname-c 2024-03-25T19:41:47.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.796+0000 7ffa267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:47.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.796+0000 7ffa267fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:41:47.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.796+0000 7ffa267fc640 1 -- 172.21.15.44:0/2416564882 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa280b44c0 con 0x7ffa281430c0 2024-03-25T19:41:47.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.796+0000 7ffa267fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:47.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.796+0000 7ffa267fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:47.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.796+0000 7ffa267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:47.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.796+0000 7ffa2d9bc640 5 monclient: authenticate success, global_id 4507 2024-03-25T19:41:47.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.796+0000 7ffa2d9bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:47.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.796+0000 7ffa267fc640 1 -- 172.21.15.44:0/2416564882 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ffa200019d0 con 0x7ffa281430c0 2024-03-25T19:41:47.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.796+0000 7ffa267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:47.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.796+0000 7ffa267fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:47.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.796+0000 7ffa267fc640 10 monclient: dump: 2024-03-25T19:41:47.797 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:47.797 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:47.797 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:47.797 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:47.798 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:47.798 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:47.798 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:47.798 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:47.798 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:47.798 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:47.798 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.796+0000 7ffa2d9bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:47.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.804+0000 7ffa267fc640 1 -- 172.21.15.44:0/2416564882 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa20001c20 con 0x7ffa281430c0 2024-03-25T19:41:47.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.804+0000 7ffa267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:47.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.804+0000 7ffa267fc640 1 -- 172.21.15.44:0/2416564882 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ffa20001fb0 con 0x7ffa281430c0 2024-03-25T19:41:47.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.804+0000 7ffa267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:47.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.804+0000 7ffa267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:47.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.804+0000 7ffa267fc640 10 monclient: dump: 2024-03-25T19:41:47.805 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:47.805 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:47.805 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:47.805 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:47.805 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:47.805 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:47.805 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:47.805 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:47.805 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:47.805 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:47.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.804+0000 7ffa2d9bc640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:47.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.804+0000 7ffa2d9bc640 4 set_mon_vals no callback set 2024-03-25T19:41:47.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.805+0000 7ffa2d9bc640 10 monclient: shutdown 2024-03-25T19:41:47.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.805+0000 7ffa2d9bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:47.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.805+0000 7ffa2d9bc640 1 -- 172.21.15.44:0/2416564882 >> v1:172.21.15.111:6789/0 conn(0x7ffa281430c0 legacy=0x7ffa2814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:47.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.805+0000 7ffa2d9bc640 1 -- 172.21.15.44:0/2416564882 shutdown_connections 2024-03-25T19:41:47.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.805+0000 7ffa2d9bc640 1 -- 172.21.15.44:0/2416564882 >> 172.21.15.44:0/2416564882 conn(0x7ffa280b7ea0 msgr2=0x7ffa280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:47.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.805+0000 7ffa2d9bc640 1 -- 172.21.15.44:0/2416564882 shutdown_connections 2024-03-25T19:41:47.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.805+0000 7ffa2d9bc640 1 -- 172.21.15.44:0/2416564882 wait complete. 2024-03-25T19:41:47.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.806+0000 7ffa2d9bc640 10 monclient: build_initial_monmap 2024-03-25T19:41:47.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.806+0000 7ffa2d9bc640 10 monclient: monmap: 2024-03-25T19:41:47.807 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:41:47.807 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:47.807 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:41:47.808 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:41:47.808 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:47.808 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:41:47.808 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:41:47.808 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:41:47.808 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:41:47.808 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:47.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.807+0000 7ffa2d9bc640 1 librados: starting msgr at 2024-03-25T19:41:47.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.807+0000 7ffa2d9bc640 1 librados: starting objecter 2024-03-25T19:41:47.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.807+0000 7ffa2d9bc640 1 Processor -- start 2024-03-25T19:41:47.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.807+0000 7ffa2d9bc640 1 -- start start 2024-03-25T19:41:47.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.807+0000 7ffa2d9bc640 1 librados: setting wanted keys 2024-03-25T19:41:47.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.807+0000 7ffa2d9bc640 1 librados: calling monclient init 2024-03-25T19:41:47.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.807+0000 7ffa2d9bc640 10 monclient: init 2024-03-25T19:41:47.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.807+0000 7ffa2d9bc640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:47.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.807+0000 7ffa2d9bc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:47.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.807+0000 7ffa2d9bc640 10 monclient(hunting): picked mon.noname-b con 0x7ffa280a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:41:47.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.807+0000 7ffa2d9bc640 10 monclient(hunting): picked mon.noname-c con 0x7ffa2813f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:41:47.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.807+0000 7ffa2d9bc640 10 monclient(hunting): picked mon.noname-a con 0x7ffa281430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:41:47.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.807+0000 7ffa2d9bc640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa280b79f0 con 0x7ffa281430c0 2024-03-25T19:41:47.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.807+0000 7ffa2d9bc640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa280b75c0 con 0x7ffa280a98f0 2024-03-25T19:41:47.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.807+0000 7ffa2d9bc640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa28135070 con 0x7ffa2813f710 2024-03-25T19:41:47.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.807+0000 7ffa2d9bc640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:47.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.807+0000 7ffa2d9bc640 10 monclient(hunting): authenticate will time out at 5771.978812s 2024-03-25T19:41:47.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.823+0000 7ffa26ffd640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ffa280a98f0 0x7ffa281d7670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:37268/0 (socket says 172.21.15.44:37268) 2024-03-25T19:41:47.823 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.823+0000 7ffa26ffd640 1 -- 172.21.15.44:0/3726271792 learned_addr learned my addr 172.21.15.44:0/3726271792 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:41:47.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.824+0000 7ffa24ff9640 10 client.?.objecter ms_handle_connect 0x7ffa280a98f0 2024-03-25T19:41:47.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.824+0000 7ffa24ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:47.829 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.829+0000 7ffa24ff9640 10 client.?.objecter ms_handle_connect 0x7ffa2813f710 2024-03-25T19:41:47.829 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.829+0000 7ffa24ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:47.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.838+0000 7ffa24ff9640 1 -- 172.21.15.44:0/3726271792 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748070483 0 0) 0x7ffa280b75c0 con 0x7ffa280a98f0 2024-03-25T19:41:47.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.839+0000 7ffa24ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:47.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.839+0000 7ffa24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:47.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.839+0000 7ffa24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:47.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.839+0000 7ffa24ff9640 10 monclient(hunting): my global_id is 4510 2024-03-25T19:41:47.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.839+0000 7ffa24ff9640 1 -- 172.21.15.44:0/3726271792 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa08004eb0 con 0x7ffa280a98f0 2024-03-25T19:41:47.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.839+0000 7ffa24ff9640 1 -- 172.21.15.44:0/3726271792 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3917021677 0 0) 0x7ffa28135070 con 0x7ffa2813f710 2024-03-25T19:41:47.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.839+0000 7ffa24ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:47.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.839+0000 7ffa24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:47.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.839+0000 7ffa24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:47.840 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.839+0000 7ffa24ff9640 10 monclient(hunting): my global_id is 4499 2024-03-25T19:41:47.840 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.839+0000 7ffa24ff9640 1 -- 172.21.15.44:0/3726271792 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa280b75c0 con 0x7ffa2813f710 2024-03-25T19:41:47.841 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.842+0000 7ffa24ff9640 10 client.?.objecter ms_handle_connect 0x7ffa281430c0 2024-03-25T19:41:47.841 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.842+0000 7ffa24ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:47.849 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.849+0000 7ffa24ff9640 1 -- 172.21.15.44:0/3726271792 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2091492393 0 0) 0x7ffa08004eb0 con 0x7ffa280a98f0 2024-03-25T19:41:47.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.849+0000 7ffa24ff9640 1 -- 172.21.15.44:0/3726271792 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa28135070 con 0x7ffa280a98f0 2024-03-25T19:41:47.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.849+0000 7ffa24ff9640 1 -- 172.21.15.44:0/3726271792 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ffa18001a10 con 0x7ffa280a98f0 2024-03-25T19:41:47.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.849+0000 7ffa24ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:47.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.849+0000 7ffa24ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:47.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.849+0000 7ffa24ff9640 10 monclient(hunting): dump: 2024-03-25T19:41:47.850 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:47.850 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:47.850 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:47.850 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:47.850 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:47.850 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:47.850 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:47.851 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:47.851 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:47.851 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:47.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.849+0000 7ffa24ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:47.862 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.862+0000 7ffa24ff9640 1 -- 172.21.15.44:0/3726271792 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 830686388 0 0) 0x7ffa28135070 con 0x7ffa280a98f0 2024-03-25T19:41:47.862 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.862+0000 7ffa24ff9640 1 -- 172.21.15.44:0/3726271792 >> v1:172.21.15.44:6790/0 conn(0x7ffa2813f710 legacy=0x7ffa281d7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:47.862 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.862+0000 7ffa24ff9640 1 -- 172.21.15.44:0/3726271792 >> v1:172.21.15.44:6789/0 conn(0x7ffa281430c0 legacy=0x7ffa281e20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:47.862 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.862+0000 7ffa24ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:41:47.862 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.862+0000 7ffa24ff9640 1 monclient: found mon.b 2024-03-25T19:41:47.863 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.862+0000 7ffa24ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:47.863 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.862+0000 7ffa24ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:41:47.863 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.862+0000 7ffa24ff9640 1 -- 172.21.15.44:0/3726271792 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa28135200 con 0x7ffa280a98f0 2024-03-25T19:41:47.863 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.862+0000 7ffa24ff9640 10 monclient: _finish_auth 0 2024-03-25T19:41:47.863 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.862+0000 7ffa24ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:41:47.863 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.862+0000 7ffa24ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:47.863 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.862+0000 7ffa2d9bc640 5 monclient: authenticate success, global_id 4510 2024-03-25T19:41:47.863 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.862+0000 7ffa2d9bc640 10 monclient: _renew_subs 2024-03-25T19:41:47.863 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.862+0000 7ffa2d9bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:41:47.863 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.862+0000 7ffa2d9bc640 1 -- 172.21.15.44:0/3726271792 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa28135910 con 0x7ffa280a98f0 2024-03-25T19:41:47.863 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.862+0000 7ffa2d9bc640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:47.863 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.862+0000 7ffa2d9bc640 10 monclient: _renew_subs 2024-03-25T19:41:47.863 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.862+0000 7ffa2d9bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:41:47.863 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.862+0000 7ffa2d9bc640 1 -- 172.21.15.44:0/3726271792 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2813b420 con 0x7ffa280a98f0 2024-03-25T19:41:47.863 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.862+0000 7ffa2d9bc640 1 librados: init done 2024-03-25T19:41:47.864 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.863+0000 7ffa2d9bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:47.864 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.863+0000 7ffa2d9bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:47.864 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.863+0000 7ffa2d9bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:41:47.864 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:47.863+0000 7ffa2d9bc640 1 -- 172.21.15.44:0/3726271792 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa280b7750 con 0x7ffa280a98f0 2024-03-25T19:41:48.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.140+0000 7ffa24ff9640 1 -- 172.21.15.44:0/3726271792 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa18001c70 con 0x7ffa280a98f0 2024-03-25T19:41:48.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.140+0000 7ffa24ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:48.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.140+0000 7ffa24ff9640 1 -- 172.21.15.44:0/3726271792 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ffa18003a60 con 0x7ffa280a98f0 2024-03-25T19:41:48.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.140+0000 7ffa24ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:48.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.140+0000 7ffa24ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:48.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.140+0000 7ffa24ff9640 10 monclient: dump: 2024-03-25T19:41:48.141 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:48.141 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:48.141 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:48.141 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:48.141 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:48.141 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:48.141 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:48.141 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:48.141 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:48.141 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:48.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.140+0000 7ffa25ffb640 4 set_mon_vals no callback set 2024-03-25T19:41:48.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.141+0000 7ffa24ff9640 1 -- 172.21.15.44:0/3726271792 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7ffa18003c80 con 0x7ffa280a98f0 2024-03-25T19:41:48.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.142+0000 7ffa1ffff640 10 monclient: get_auth_request con 0x7ffa0806ddd0 auth_method 0 2024-03-25T19:41:48.142 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.142+0000 7ffa24ff9640 10 client.4510.objecter ms_handle_connect 0x7ffa0806ddd0 2024-03-25T19:41:48.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.144+0000 7ffa24ff9640 1 -- 172.21.15.44:0/3726271792 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7ffa180547d0 con 0x7ffa280a98f0 2024-03-25T19:41:48.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.144+0000 7ffa24ff9640 10 client.4510.objecter ms_dispatch 0x7ffa28000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:48.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.144+0000 7ffa24ff9640 3 client.4510.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:48.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.144+0000 7ffa24ff9640 3 client.4510.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:48.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.144+0000 7ffa24ff9640 20 client.4510.objecter dump_active .. 0 homeless 2024-03-25T19:41:48.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.161+0000 7ffa24ff9640 1 -- 172.21.15.44:0/3726271792 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffa180502e0 con 0x7ffa280a98f0 2024-03-25T19:41:48.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.161+0000 7ffa24ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:48.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.161+0000 7ffa24ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:48.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.424+0000 7ffa2d9bc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:48.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.424+0000 7ffa2d9bc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:48.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.424+0000 7ffa2d9bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:41:48.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.424+0000 7ffa2d9bc640 1 -- 172.21.15.44:0/3726271792 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa280627b0 con 0x7ffa280a98f0 2024-03-25T19:41:48.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.467+0000 7ffa24ff9640 1 -- 172.21.15.44:0/3726271792 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7ffa18052210 con 0x7ffa280a98f0 2024-03-25T19:41:48.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.467+0000 7ffa24ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:48.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.467+0000 7ffa24ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:48.467 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:41:48.468 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:48.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.470+0000 7ffa2d9bc640 10 librados: watch_flush enter 2024-03-25T19:41:48.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.470+0000 7ffa2d9bc640 10 librados: watch_flush exit 2024-03-25T19:41:48.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.470+0000 7ffa2d9bc640 20 client.4510.objecter shutdown clearing up homeless session... 2024-03-25T19:41:48.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.470+0000 7ffa2d9bc640 10 client.4510.objecter successfully canceled tick 2024-03-25T19:41:48.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.470+0000 7ffa2d9bc640 1 -- 172.21.15.44:0/3726271792 >> v1:172.21.15.111:6800/3883215793 conn(0x7ffa0806ddd0 legacy=0x7ffa08070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:48.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.470+0000 7ffa2d9bc640 10 monclient: shutdown 2024-03-25T19:41:48.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.470+0000 7ffa2d9bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:48.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.470+0000 7ffa2d9bc640 1 -- 172.21.15.44:0/3726271792 >> v1:172.21.15.111:6789/0 conn(0x7ffa280a98f0 legacy=0x7ffa281d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:48.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.470+0000 7ffa2d9bc640 1 -- 172.21.15.44:0/3726271792 shutdown_connections 2024-03-25T19:41:48.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.470+0000 7ffa2d9bc640 1 -- 172.21.15.44:0/3726271792 >> 172.21.15.44:0/3726271792 conn(0x7ffa280b7ea0 msgr2=0x7ffa28074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:48.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.470+0000 7ffa2d9bc640 1 -- 172.21.15.44:0/3726271792 shutdown_connections 2024-03-25T19:41:48.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.470+0000 7ffa2d9bc640 1 -- 172.21.15.44:0/3726271792 wait complete. 2024-03-25T19:41:48.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:48.470+0000 7ffa2d9bc640 1 librados: shutdown 2024-03-25T19:41:48.483 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:55.290 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:55.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.410+0000 7f6fe75d6640 10 monclient: get_monmap_and_config 2024-03-25T19:41:55.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.410+0000 7f6fe75d6640 10 monclient: build_initial_monmap 2024-03-25T19:41:55.411 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.411+0000 7f6fe75d6640 10 monclient: monmap: 2024-03-25T19:41:55.411 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:41:55.411 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:55.411 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:41:55.412134+0000 2024-03-25T19:41:55.411 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:41:55.412134+0000 2024-03-25T19:41:55.411 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:55.411 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:55.411 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:41:55.411 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:41:55.411 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:41:55.412 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:55.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.411+0000 7f6fe75d6640 1 Processor -- start 2024-03-25T19:41:55.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.411+0000 7f6fe75d6640 1 -- start start 2024-03-25T19:41:55.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.411+0000 7f6fe75d6640 10 monclient: init 2024-03-25T19:41:55.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.412+0000 7f6fe75d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:55.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.412+0000 7f6fe75d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:55.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.412+0000 7f6fe75d6640 10 monclient(hunting): picked mon.noname-a con 0x7f6fe00a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:41:55.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.412+0000 7f6fe75d6640 10 monclient(hunting): picked mon.noname-c con 0x7f6fe013f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:41:55.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.412+0000 7f6fe75d6640 10 monclient(hunting): picked mon.noname-b con 0x7f6fe01430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:41:55.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.412+0000 7f6fe75d6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe0142520 con 0x7f6fe00a98f0 2024-03-25T19:41:55.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.412+0000 7f6fe75d6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe00b28a0 con 0x7f6fe013f710 2024-03-25T19:41:55.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.412+0000 7f6fe75d6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe00b2a30 con 0x7f6fe01430c0 2024-03-25T19:41:55.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.412+0000 7f6fe75d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:55.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.412+0000 7f6fe75d6640 10 monclient(hunting): authenticate will time out at 5779.583600s 2024-03-25T19:41:55.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.419+0000 7f6fe5372640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f6fe00a98f0 0x7f6fe00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42786/0 (socket says 172.21.15.44:42786) 2024-03-25T19:41:55.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.419+0000 7f6fe5372640 1 -- 172.21.15.44:0/3484871196 learned_addr learned my addr 172.21.15.44:0/3484871196 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:41:55.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.435+0000 7f6fd7fff640 1 -- 172.21.15.44:0/3484871196 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3090256520 0 0) 0x7f6fe0142520 con 0x7f6fe00a98f0 2024-03-25T19:41:55.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.435+0000 7f6fd7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:55.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.435+0000 7f6fd7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:55.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.435+0000 7f6fd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:55.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.435+0000 7f6fd7fff640 10 monclient(hunting): my global_id is 4527 2024-03-25T19:41:55.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.435+0000 7f6fd7fff640 1 -- 172.21.15.44:0/3484871196 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fcc003520 con 0x7f6fe00a98f0 2024-03-25T19:41:55.441 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.441+0000 7f6fd7fff640 1 -- 172.21.15.44:0/3484871196 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3840528439 0 0) 0x7f6fcc003520 con 0x7f6fe00a98f0 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.441+0000 7f6fd7fff640 1 -- 172.21.15.44:0/3484871196 >> v1:172.21.15.44:6790/0 conn(0x7f6fe01430c0 legacy=0x7f6fe014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.441+0000 7f6fd7fff640 1 -- 172.21.15.44:0/3484871196 >> v1:172.21.15.111:6789/0 conn(0x7f6fe013f710 legacy=0x7f6fe01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.441+0000 7f6fd7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.441+0000 7f6fd7fff640 1 monclient: found mon.noname-a 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.441+0000 7f6fd7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.441+0000 7f6fd7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.441+0000 7f6fd7fff640 1 -- 172.21.15.44:0/3484871196 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe00b44c0 con 0x7f6fe00a98f0 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.441+0000 7f6fd7fff640 10 monclient: _finish_auth 0 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.441+0000 7f6fd7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.441+0000 7f6fd7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.441+0000 7f6fd7fff640 1 -- 172.21.15.44:0/3484871196 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6fdc001690 con 0x7f6fe00a98f0 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.441+0000 7f6fe75d6640 5 monclient: authenticate success, global_id 4527 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.441+0000 7f6fe75d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:55.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.441+0000 7f6fd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:55.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.441+0000 7f6fd7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:55.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.441+0000 7f6fd7fff640 10 monclient: dump: 2024-03-25T19:41:55.443 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:55.443 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:55.443 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:55.443 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:55.443 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:55.443 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:55.443 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:55.443 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:55.443 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:55.443 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:55.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.441+0000 7f6fe75d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:55.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.455+0000 7f6fd7fff640 1 -- 172.21.15.44:0/3484871196 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fdc0017f0 con 0x7f6fe00a98f0 2024-03-25T19:41:55.455 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.455+0000 7f6fd7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:55.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.455+0000 7f6fd7fff640 1 -- 172.21.15.44:0/3484871196 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6fdc001c00 con 0x7f6fe00a98f0 2024-03-25T19:41:55.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.455+0000 7f6fd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:55.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.455+0000 7f6fd7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:55.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.455+0000 7f6fd7fff640 10 monclient: dump: 2024-03-25T19:41:55.456 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:55.456 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:55.456 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:55.456 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:55.456 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:55.456 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:55.456 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:55.456 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:55.456 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:55.456 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:55.456 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.455+0000 7f6fe75d6640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:55.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.455+0000 7f6fe75d6640 4 set_mon_vals no callback set 2024-03-25T19:41:55.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.456+0000 7f6fe75d6640 10 monclient: shutdown 2024-03-25T19:41:55.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.456+0000 7f6fe75d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:55.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.456+0000 7f6fe75d6640 1 -- 172.21.15.44:0/3484871196 >> v1:172.21.15.44:6789/0 conn(0x7f6fe00a98f0 legacy=0x7f6fe00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:55.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.456+0000 7f6fe75d6640 1 -- 172.21.15.44:0/3484871196 shutdown_connections 2024-03-25T19:41:55.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.456+0000 7f6fe75d6640 1 -- 172.21.15.44:0/3484871196 >> 172.21.15.44:0/3484871196 conn(0x7f6fe00b7ea0 msgr2=0x7f6fe00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:55.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.456+0000 7f6fe75d6640 1 -- 172.21.15.44:0/3484871196 shutdown_connections 2024-03-25T19:41:55.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.456+0000 7f6fe75d6640 1 -- 172.21.15.44:0/3484871196 wait complete. 2024-03-25T19:41:55.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.457+0000 7f6fe75d6640 10 monclient: build_initial_monmap 2024-03-25T19:41:55.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.457+0000 7f6fe75d6640 10 monclient: monmap: 2024-03-25T19:41:55.459 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:41:55.459 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:55.459 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:41:55.459 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:41:55.459 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:55.460 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:41:55.460 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:41:55.460 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:41:55.460 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:41:55.460 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:55.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.457+0000 7f6fe75d6640 1 librados: starting msgr at 2024-03-25T19:41:55.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.457+0000 7f6fe75d6640 1 librados: starting objecter 2024-03-25T19:41:55.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.457+0000 7f6fe75d6640 1 Processor -- start 2024-03-25T19:41:55.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.457+0000 7f6fe75d6640 1 -- start start 2024-03-25T19:41:55.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.457+0000 7f6fe75d6640 1 librados: setting wanted keys 2024-03-25T19:41:55.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.457+0000 7f6fe75d6640 1 librados: calling monclient init 2024-03-25T19:41:55.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.457+0000 7f6fe75d6640 10 monclient: init 2024-03-25T19:41:55.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.458+0000 7f6fe75d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:55.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.458+0000 7f6fe75d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:55.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.458+0000 7f6fe75d6640 10 monclient(hunting): picked mon.noname-a con 0x7f6fe00a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:41:55.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.458+0000 7f6fe75d6640 10 monclient(hunting): picked mon.noname-b con 0x7f6fe013f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:41:55.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.458+0000 7f6fe75d6640 10 monclient(hunting): picked mon.noname-c con 0x7f6fe01430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:41:55.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.458+0000 7f6fe75d6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe0135ac0 con 0x7f6fe00a98f0 2024-03-25T19:41:55.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.458+0000 7f6fe75d6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe0137030 con 0x7f6fe013f710 2024-03-25T19:41:55.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.458+0000 7f6fe75d6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe01372a0 con 0x7f6fe01430c0 2024-03-25T19:41:55.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.458+0000 7f6fe75d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:55.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.458+0000 7f6fe75d6640 10 monclient(hunting): authenticate will time out at 5779.629253s 2024-03-25T19:41:55.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.458+0000 7f6fe4b71640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f6fe013f710 0x7f6fe01379a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:56950/0 (socket says 172.21.15.44:56950) 2024-03-25T19:41:55.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.458+0000 7f6fe4b71640 1 -- 172.21.15.44:0/3367306337 learned_addr learned my addr 172.21.15.44:0/3367306337 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:41:55.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.459+0000 7f6fd5ffb640 10 client.?.objecter ms_handle_connect 0x7f6fe013f710 2024-03-25T19:41:55.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.459+0000 7f6fd5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:55.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.459+0000 7f6fd5ffb640 10 client.?.objecter ms_handle_connect 0x7f6fe01430c0 2024-03-25T19:41:55.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.459+0000 7f6fd5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:55.517 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.517+0000 7f6fd5ffb640 10 client.?.objecter ms_handle_connect 0x7f6fe00a98f0 2024-03-25T19:41:55.517 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.517+0000 7f6fd5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:55.522 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.522+0000 7f6fd5ffb640 1 -- 172.21.15.44:0/3367306337 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809122324 0 0) 0x7f6fe0135ac0 con 0x7f6fe00a98f0 2024-03-25T19:41:55.522 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.522+0000 7f6fd5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:55.522 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.522+0000 7f6fd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:55.522 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.522+0000 7f6fd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:55.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.522+0000 7f6fd5ffb640 10 monclient(hunting): my global_id is 4530 2024-03-25T19:41:55.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.522+0000 7f6fd5ffb640 1 -- 172.21.15.44:0/3367306337 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fc4004eb0 con 0x7f6fe00a98f0 2024-03-25T19:41:55.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.531+0000 7f6fd5ffb640 1 -- 172.21.15.44:0/3367306337 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881496235 0 0) 0x7f6fe0137030 con 0x7f6fe013f710 2024-03-25T19:41:55.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.531+0000 7f6fd5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:55.531 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.531+0000 7f6fd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:55.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.531+0000 7f6fd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:55.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.531+0000 7f6fd5ffb640 10 monclient(hunting): my global_id is 4513 2024-03-25T19:41:55.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.531+0000 7f6fd5ffb640 1 -- 172.21.15.44:0/3367306337 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe0135ac0 con 0x7f6fe013f710 2024-03-25T19:41:55.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.533+0000 7f6fd5ffb640 1 -- 172.21.15.44:0/3367306337 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2432106584 0 0) 0x7f6fc4004eb0 con 0x7f6fe00a98f0 2024-03-25T19:41:55.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.534+0000 7f6fd5ffb640 1 -- 172.21.15.44:0/3367306337 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fe0137030 con 0x7f6fe00a98f0 2024-03-25T19:41:55.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.534+0000 7f6fd5ffb640 1 -- 172.21.15.44:0/3367306337 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6fdc001bc0 con 0x7f6fe00a98f0 2024-03-25T19:41:55.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.534+0000 7f6fd5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:55.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.534+0000 7f6fd5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:55.534 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.534+0000 7f6fd5ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:55.534 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:55.534 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:55.534 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:55.534 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:55.534 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:55.534 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:55.535 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:55.535 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:55.535 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:55.535 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:55.535 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.534+0000 7f6fd5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:55.535 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.535+0000 7f6fd5ffb640 1 -- 172.21.15.44:0/3367306337 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3780874068 0 0) 0x7f6fe01372a0 con 0x7f6fe01430c0 2024-03-25T19:41:55.535 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.535+0000 7f6fd5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:55.536 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.535+0000 7f6fd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:55.536 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.535+0000 7f6fd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:55.536 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.535+0000 7f6fd5ffb640 10 monclient(hunting): my global_id is 4502 2024-03-25T19:41:55.536 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.535+0000 7f6fd5ffb640 1 -- 172.21.15.44:0/3367306337 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fc4004eb0 con 0x7f6fe01430c0 2024-03-25T19:41:55.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.547+0000 7f6fd5ffb640 1 -- 172.21.15.44:0/3367306337 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1571560166 0 0) 0x7f6fc4004eb0 con 0x7f6fe01430c0 2024-03-25T19:41:55.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.547+0000 7f6fd5ffb640 1 -- 172.21.15.44:0/3367306337 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fe01372a0 con 0x7f6fe01430c0 2024-03-25T19:41:55.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.547+0000 7f6fd5ffb640 1 -- 172.21.15.44:0/3367306337 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6fd8001a60 con 0x7f6fe01430c0 2024-03-25T19:41:55.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.547+0000 7f6fd5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:55.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.547+0000 7f6fd5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:55.547 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.547+0000 7f6fd5ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:55.547 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:55.547 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:55.547 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:55.547 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:55.547 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:55.547 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:55.548 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:55.548 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:55.548 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:55.548 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:55.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.548+0000 7f6fd5ffb640 1 -- 172.21.15.44:0/3367306337 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1675470258 0 0) 0x7f6fe0135ac0 con 0x7f6fe013f710 2024-03-25T19:41:55.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.548+0000 7f6fd5ffb640 1 -- 172.21.15.44:0/3367306337 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fc4004eb0 con 0x7f6fe013f710 2024-03-25T19:41:55.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.548+0000 7f6fd5ffb640 1 -- 172.21.15.44:0/3367306337 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6fd00019b0 con 0x7f6fe013f710 2024-03-25T19:41:55.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.548+0000 7f6fd5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:55.549 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.548+0000 7f6fd5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:55.549 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.548+0000 7f6fd5ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:55.549 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:55.549 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:55.549 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:55.549 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:55.549 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:55.549 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:55.549 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:55.549 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:55.549 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:55.549 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:55.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.552+0000 7f6fd5ffb640 1 -- 172.21.15.44:0/3367306337 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3616838996 0 0) 0x7f6fe0137030 con 0x7f6fe00a98f0 2024-03-25T19:41:55.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.552+0000 7f6fd5ffb640 1 -- 172.21.15.44:0/3367306337 >> v1:172.21.15.44:6790/0 conn(0x7f6fe01430c0 legacy=0x7f6fe01e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:55.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.552+0000 7f6fd5ffb640 1 -- 172.21.15.44:0/3367306337 >> v1:172.21.15.111:6789/0 conn(0x7f6fe013f710 legacy=0x7f6fe01379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:55.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.552+0000 7f6fd5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:41:55.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.552+0000 7f6fd5ffb640 1 monclient: found mon.a 2024-03-25T19:41:55.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.552+0000 7f6fd5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:55.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.552+0000 7f6fd5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:41:55.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.552+0000 7f6fd5ffb640 1 -- 172.21.15.44:0/3367306337 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe0137580 con 0x7f6fe00a98f0 2024-03-25T19:41:55.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.552+0000 7f6fd5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:41:55.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.552+0000 7f6fd5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:41:55.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.552+0000 7f6fd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:55.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.552+0000 7f6fe75d6640 5 monclient: authenticate success, global_id 4530 2024-03-25T19:41:55.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.552+0000 7f6fe75d6640 10 monclient: _renew_subs 2024-03-25T19:41:55.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.552+0000 7f6fe75d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:41:55.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.552+0000 7f6fe75d6640 1 -- 172.21.15.44:0/3367306337 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe01363b0 con 0x7f6fe00a98f0 2024-03-25T19:41:55.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.552+0000 7f6fe75d6640 10 client.4530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:55.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.552+0000 7f6fe75d6640 10 monclient: _renew_subs 2024-03-25T19:41:55.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.552+0000 7f6fe75d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:41:55.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.552+0000 7f6fe75d6640 1 -- 172.21.15.44:0/3367306337 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe01353a0 con 0x7f6fe00a98f0 2024-03-25T19:41:55.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.552+0000 7f6fe75d6640 1 librados: init done 2024-03-25T19:41:55.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.553+0000 7f6fe75d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:55.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.553+0000 7f6fe75d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:55.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.553+0000 7f6fe75d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:41:55.554 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.553+0000 7f6fe75d6640 1 -- 172.21.15.44:0/3367306337 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe0074cc0 con 0x7f6fe00a98f0 2024-03-25T19:41:55.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.771+0000 7f6fd5ffb640 1 -- 172.21.15.44:0/3367306337 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fdc004a90 con 0x7f6fe00a98f0 2024-03-25T19:41:55.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.771+0000 7f6fd5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:55.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.771+0000 7f6fd5ffb640 1 -- 172.21.15.44:0/3367306337 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6fdc001250 con 0x7f6fe00a98f0 2024-03-25T19:41:55.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.771+0000 7f6fd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:55.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.771+0000 7f6fd5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:55.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.771+0000 7f6fd5ffb640 10 monclient: dump: 2024-03-25T19:41:55.771 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:41:55.771 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:41:55.772 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:55.772 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:41:55.772 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:55.772 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:41:55.772 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:41:55.772 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:41:55.772 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:41:55.772 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:41:55.772 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.771+0000 7f6fd77fe640 4 set_mon_vals no callback set 2024-03-25T19:41:55.865 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.865+0000 7f6fd5ffb640 1 -- 172.21.15.44:0/3367306337 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f6fdc01ac10 con 0x7f6fe00a98f0 2024-03-25T19:41:55.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.866+0000 7f6fe4b71640 10 monclient: get_auth_request con 0x7f6fc406e160 auth_method 0 2024-03-25T19:41:55.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.866+0000 7f6fd5ffb640 10 client.4530.objecter ms_handle_connect 0x7f6fc406e160 2024-03-25T19:41:55.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.870+0000 7f6fd5ffb640 1 -- 172.21.15.44:0/3367306337 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f6fdc054870 con 0x7f6fe00a98f0 2024-03-25T19:41:55.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.871+0000 7f6fd5ffb640 10 client.4530.objecter ms_dispatch 0x7f6fe0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:55.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.871+0000 7f6fd5ffb640 3 client.4530.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:55.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.871+0000 7f6fd5ffb640 3 client.4530.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:55.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.871+0000 7f6fd5ffb640 20 client.4530.objecter dump_active .. 0 homeless 2024-03-25T19:41:55.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.871+0000 7f6fd5ffb640 1 -- 172.21.15.44:0/3367306337 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6fdc0501b0 con 0x7f6fe00a98f0 2024-03-25T19:41:55.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.871+0000 7f6fd5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:55.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:55.871+0000 7f6fd5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:56.127 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:56.127+0000 7f6fe75d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:56.127 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:56.127+0000 7f6fe75d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:56.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:56.127+0000 7f6fe75d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:41:56.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:56.127+0000 7f6fe75d6640 1 -- 172.21.15.44:0/3367306337 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6fe00627b0 con 0x7f6fe00a98f0 2024-03-25T19:41:56.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:56.146+0000 7f6fd5ffb640 1 -- 172.21.15.44:0/3367306337 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f6fdc0522b0 con 0x7f6fe00a98f0 2024-03-25T19:41:56.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:56.146+0000 7f6fd5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:56.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:56.146+0000 7f6fd5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:56.146 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:41:56.146 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:56.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:56.148+0000 7f6fe75d6640 10 librados: watch_flush enter 2024-03-25T19:41:56.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:56.148+0000 7f6fe75d6640 10 librados: watch_flush exit 2024-03-25T19:41:56.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:56.148+0000 7f6fe75d6640 20 client.4530.objecter shutdown clearing up homeless session... 2024-03-25T19:41:56.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:56.148+0000 7f6fe75d6640 10 client.4530.objecter successfully canceled tick 2024-03-25T19:41:56.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:56.148+0000 7f6fe75d6640 1 -- 172.21.15.44:0/3367306337 >> v1:172.21.15.111:6800/3883215793 conn(0x7f6fc406e160 legacy=0x7f6fc4070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:56.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:56.148+0000 7f6fe75d6640 10 monclient: shutdown 2024-03-25T19:41:56.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:56.148+0000 7f6fe75d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:56.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:56.148+0000 7f6fe75d6640 1 -- 172.21.15.44:0/3367306337 >> v1:172.21.15.44:6789/0 conn(0x7f6fe00a98f0 legacy=0x7f6fe0133440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:56.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:56.148+0000 7f6fe75d6640 1 -- 172.21.15.44:0/3367306337 shutdown_connections 2024-03-25T19:41:56.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:56.148+0000 7f6fe75d6640 1 -- 172.21.15.44:0/3367306337 >> 172.21.15.44:0/3367306337 conn(0x7f6fe00b7ea0 msgr2=0x7f6fe0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:56.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:56.148+0000 7f6fe75d6640 1 -- 172.21.15.44:0/3367306337 shutdown_connections 2024-03-25T19:41:56.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:56.148+0000 7f6fe75d6640 1 -- 172.21.15.44:0/3367306337 wait complete. 2024-03-25T19:41:56.150 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:41:56.149+0000 7f6fe75d6640 1 librados: shutdown 2024-03-25T19:41:56.160 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:59.788 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:06:50.690 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:59.795 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:06:50.697 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:59.831 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:06:50.733 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:59.862 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:06:50.742 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:59.865 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:06:50.745 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:59.887 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:06:50.789 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:59.893 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:06:50.795 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:59.895 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:06:50.774 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:59.937 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:06:50.839 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:02.967 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:03.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.083+0000 7f3a4f614640 10 monclient: get_monmap_and_config 2024-03-25T19:42:03.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.083+0000 7f3a4f614640 10 monclient: build_initial_monmap 2024-03-25T19:42:03.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.083+0000 7f3a4f614640 10 monclient: monmap: 2024-03-25T19:42:03.084 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:42:03.084 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:03.084 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:42:03.084651+0000 2024-03-25T19:42:03.084 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:42:03.084651+0000 2024-03-25T19:42:03.084 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:03.084 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:03.084 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:42:03.084 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:42:03.084 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:42:03.084 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:03.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.084+0000 7f3a4f614640 1 Processor -- start 2024-03-25T19:42:03.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.084+0000 7f3a4f614640 1 -- start start 2024-03-25T19:42:03.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.084+0000 7f3a4f614640 10 monclient: init 2024-03-25T19:42:03.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.085+0000 7f3a4f614640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:03.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.085+0000 7f3a4f614640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:03.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.085+0000 7f3a4f614640 10 monclient(hunting): picked mon.noname-c con 0x7f3a480a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:42:03.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.085+0000 7f3a4f614640 10 monclient(hunting): picked mon.noname-a con 0x7f3a4813f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:42:03.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.085+0000 7f3a4f614640 10 monclient(hunting): picked mon.noname-b con 0x7f3a481430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:42:03.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.085+0000 7f3a4f614640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a48142520 con 0x7f3a4813f710 2024-03-25T19:42:03.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.085+0000 7f3a4f614640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a480b28a0 con 0x7f3a480a98f0 2024-03-25T19:42:03.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.085+0000 7f3a4f614640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a480b2a30 con 0x7f3a481430c0 2024-03-25T19:42:03.086 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.085+0000 7f3a4f614640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:03.087 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.085+0000 7f3a4f614640 10 monclient(hunting): authenticate will time out at 5787.256566s 2024-03-25T19:42:03.089 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.089+0000 7f3a4cbaf640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f3a4813f710 0x7f3a481429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:44328/0 (socket says 172.21.15.44:44328) 2024-03-25T19:42:03.089 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.089+0000 7f3a4cbaf640 1 -- 172.21.15.44:0/3622333430 learned_addr learned my addr 172.21.15.44:0/3622333430 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:42:03.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.101+0000 7f3a3ffff640 1 -- 172.21.15.44:0/3622333430 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2531246774 0 0) 0x7f3a48142520 con 0x7f3a4813f710 2024-03-25T19:42:03.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.101+0000 7f3a3ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:03.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.101+0000 7f3a3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:03.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.101+0000 7f3a3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:03.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.101+0000 7f3a3ffff640 10 monclient(hunting): my global_id is 4533 2024-03-25T19:42:03.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.101+0000 7f3a3ffff640 1 -- 172.21.15.44:0/3622333430 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a34003520 con 0x7f3a4813f710 2024-03-25T19:42:03.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.113+0000 7f3a3ffff640 1 -- 172.21.15.44:0/3622333430 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2472808571 0 0) 0x7f3a34003520 con 0x7f3a4813f710 2024-03-25T19:42:03.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.113+0000 7f3a3ffff640 1 -- 172.21.15.44:0/3622333430 >> v1:172.21.15.44:6790/0 conn(0x7f3a481430c0 legacy=0x7f3a4814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.113+0000 7f3a3ffff640 1 -- 172.21.15.44:0/3622333430 >> v1:172.21.15.111:6789/0 conn(0x7f3a480a98f0 legacy=0x7f3a480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.113+0000 7f3a3ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:03.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.113+0000 7f3a3ffff640 1 monclient: found mon.noname-a 2024-03-25T19:42:03.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.113+0000 7f3a3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:03.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.113+0000 7f3a3ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:42:03.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.113+0000 7f3a3ffff640 1 -- 172.21.15.44:0/3622333430 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a480b44c0 con 0x7f3a4813f710 2024-03-25T19:42:03.113 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.113+0000 7f3a3ffff640 10 monclient: _finish_auth 0 2024-03-25T19:42:03.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.113+0000 7f3a3ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:03.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.113+0000 7f3a3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:03.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.113+0000 7f3a3ffff640 1 -- 172.21.15.44:0/3622333430 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3a38001690 con 0x7f3a4813f710 2024-03-25T19:42:03.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.113+0000 7f3a3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:03.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.113+0000 7f3a3ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:03.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.113+0000 7f3a3ffff640 10 monclient: dump: 2024-03-25T19:42:03.114 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:03.114 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:03.114 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:03.114 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:03.114 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:03.114 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:03.114 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:03.114 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:03.115 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:03.115 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:03.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.113+0000 7f3a4f614640 5 monclient: authenticate success, global_id 4533 2024-03-25T19:42:03.115 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.113+0000 7f3a4f614640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:03.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.116+0000 7f3a3ffff640 1 -- 172.21.15.44:0/3622333430 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a380017f0 con 0x7f3a4813f710 2024-03-25T19:42:03.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.116+0000 7f3a3ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:03.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.116+0000 7f3a3ffff640 1 -- 172.21.15.44:0/3622333430 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3a38001c00 con 0x7f3a4813f710 2024-03-25T19:42:03.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.116+0000 7f3a3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:03.116 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.116+0000 7f3a3ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:03.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.116+0000 7f3a3ffff640 10 monclient: dump: 2024-03-25T19:42:03.117 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:03.117 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:03.117 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:03.117 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:03.117 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:03.117 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:03.117 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:03.117 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:03.117 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:03.117 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:03.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.116+0000 7f3a4f614640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:03.117 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.116+0000 7f3a4f614640 4 set_mon_vals no callback set 2024-03-25T19:42:03.119 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.117+0000 7f3a4f614640 10 monclient: shutdown 2024-03-25T19:42:03.119 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.117+0000 7f3a4f614640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.117+0000 7f3a4f614640 1 -- 172.21.15.44:0/3622333430 >> v1:172.21.15.44:6789/0 conn(0x7f3a4813f710 legacy=0x7f3a481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.117+0000 7f3a4f614640 1 -- 172.21.15.44:0/3622333430 shutdown_connections 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.117+0000 7f3a4f614640 1 -- 172.21.15.44:0/3622333430 >> 172.21.15.44:0/3622333430 conn(0x7f3a480b7ea0 msgr2=0x7f3a480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.117+0000 7f3a4f614640 1 -- 172.21.15.44:0/3622333430 shutdown_connections 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.117+0000 7f3a4f614640 1 -- 172.21.15.44:0/3622333430 wait complete. 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.118+0000 7f3a4f614640 10 monclient: build_initial_monmap 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.118+0000 7f3a4f614640 10 monclient: monmap: 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:42:03.120 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:03.121 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:42:03.121 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:42:03.121 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:42:03.121 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:42:03.121 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:03.121 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.118+0000 7f3a4f614640 1 librados: starting msgr at 2024-03-25T19:42:03.121 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.118+0000 7f3a4f614640 1 librados: starting objecter 2024-03-25T19:42:03.121 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.118+0000 7f3a4f614640 1 Processor -- start 2024-03-25T19:42:03.121 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.118+0000 7f3a4f614640 1 -- start start 2024-03-25T19:42:03.121 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.118+0000 7f3a4f614640 1 librados: setting wanted keys 2024-03-25T19:42:03.121 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.118+0000 7f3a4f614640 1 librados: calling monclient init 2024-03-25T19:42:03.121 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.118+0000 7f3a4f614640 10 monclient: init 2024-03-25T19:42:03.121 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.119+0000 7f3a4f614640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:03.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.119+0000 7f3a4f614640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:03.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.119+0000 7f3a4f614640 10 monclient(hunting): picked mon.noname-c con 0x7f3a480a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T19:42:03.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.119+0000 7f3a4f614640 10 monclient(hunting): picked mon.noname-b con 0x7f3a4813f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:42:03.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.119+0000 7f3a4f614640 10 monclient(hunting): picked mon.noname-a con 0x7f3a481430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:42:03.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.119+0000 7f3a4f614640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a480abd70 con 0x7f3a481430c0 2024-03-25T19:42:03.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.119+0000 7f3a4f614640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a481d7710 con 0x7f3a4813f710 2024-03-25T19:42:03.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.119+0000 7f3a4f614640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a481d8340 con 0x7f3a480a98f0 2024-03-25T19:42:03.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.119+0000 7f3a4f614640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:03.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.119+0000 7f3a4f614640 10 monclient(hunting): authenticate will time out at 5787.290668s 2024-03-25T19:42:03.125 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.125+0000 7f3a4cbaf640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3a4813f710 0x7f3a481d7c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:50926/0 (socket says 172.21.15.44:50926) 2024-03-25T19:42:03.125 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.125+0000 7f3a4cbaf640 1 -- 172.21.15.44:0/1920957045 learned_addr learned my addr 172.21.15.44:0/1920957045 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:42:03.125 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.126+0000 7f3a3dffb640 10 client.?.objecter ms_handle_connect 0x7f3a4813f710 2024-03-25T19:42:03.126 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.126+0000 7f3a3dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:03.132 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.133+0000 7f3a3dffb640 10 client.?.objecter ms_handle_connect 0x7f3a481430c0 2024-03-25T19:42:03.132 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.133+0000 7f3a3dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:03.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.134+0000 7f3a3dffb640 10 client.?.objecter ms_handle_connect 0x7f3a480a98f0 2024-03-25T19:42:03.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.134+0000 7f3a3dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:03.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.134+0000 7f3a3dffb640 1 -- 172.21.15.44:0/1920957045 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 522235636 0 0) 0x7f3a481d7710 con 0x7f3a4813f710 2024-03-25T19:42:03.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.134+0000 7f3a3dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:03.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.134+0000 7f3a3dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:03.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.134+0000 7f3a3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:03.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.134+0000 7f3a3dffb640 10 monclient(hunting): my global_id is 4519 2024-03-25T19:42:03.134 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.134+0000 7f3a3dffb640 1 -- 172.21.15.44:0/1920957045 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a2c004eb0 con 0x7f3a4813f710 2024-03-25T19:42:03.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.139+0000 7f3a3dffb640 1 -- 172.21.15.44:0/1920957045 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 73701692 0 0) 0x7f3a480abd70 con 0x7f3a481430c0 2024-03-25T19:42:03.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.139+0000 7f3a3dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:03.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.139+0000 7f3a3dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:03.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.139+0000 7f3a3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:03.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.139+0000 7f3a3dffb640 10 monclient(hunting): my global_id is 4536 2024-03-25T19:42:03.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.139+0000 7f3a3dffb640 1 -- 172.21.15.44:0/1920957045 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a481d7710 con 0x7f3a481430c0 2024-03-25T19:42:03.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.141+0000 7f3a3dffb640 1 -- 172.21.15.44:0/1920957045 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 311032289 0 0) 0x7f3a481d8340 con 0x7f3a480a98f0 2024-03-25T19:42:03.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.141+0000 7f3a3dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:03.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.141+0000 7f3a3dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:03.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.141+0000 7f3a3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:03.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.141+0000 7f3a3dffb640 10 monclient(hunting): my global_id is 4508 2024-03-25T19:42:03.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.141+0000 7f3a3dffb640 1 -- 172.21.15.44:0/1920957045 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a480abd70 con 0x7f3a480a98f0 2024-03-25T19:42:03.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.144+0000 7f3a3dffb640 1 -- 172.21.15.44:0/1920957045 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3715372507 0 0) 0x7f3a481d7710 con 0x7f3a481430c0 2024-03-25T19:42:03.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.144+0000 7f3a3dffb640 1 -- 172.21.15.44:0/1920957045 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a481d8340 con 0x7f3a481430c0 2024-03-25T19:42:03.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.144+0000 7f3a3dffb640 1 -- 172.21.15.44:0/1920957045 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3a40001b00 con 0x7f3a481430c0 2024-03-25T19:42:03.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.144+0000 7f3a3dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:03.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.144+0000 7f3a3dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:03.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.144+0000 7f3a3dffb640 10 monclient(hunting): dump: 2024-03-25T19:42:03.145 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:03.145 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:03.145 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:03.145 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:03.145 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:03.145 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:03.145 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:03.145 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:03.146 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:03.146 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:03.146 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.144+0000 7f3a3dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:03.148 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.148+0000 7f3a3dffb640 1 -- 172.21.15.44:0/1920957045 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1647458580 0 0) 0x7f3a2c004eb0 con 0x7f3a4813f710 2024-03-25T19:42:03.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.148+0000 7f3a3dffb640 1 -- 172.21.15.44:0/1920957045 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a481d7710 con 0x7f3a4813f710 2024-03-25T19:42:03.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.148+0000 7f3a3dffb640 1 -- 172.21.15.44:0/1920957045 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3a38001c00 con 0x7f3a4813f710 2024-03-25T19:42:03.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.148+0000 7f3a3dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:03.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.148+0000 7f3a3dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:03.149 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.148+0000 7f3a3dffb640 10 monclient(hunting): dump: 2024-03-25T19:42:03.149 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:03.149 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:03.149 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:03.149 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:03.149 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:03.149 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:03.149 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:03.149 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:03.150 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:03.150 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:03.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.154+0000 7f3a3dffb640 1 -- 172.21.15.44:0/1920957045 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3966753039 0 0) 0x7f3a481d8340 con 0x7f3a481430c0 2024-03-25T19:42:03.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.154+0000 7f3a3dffb640 1 -- 172.21.15.44:0/1920957045 >> v1:172.21.15.44:6790/0 conn(0x7f3a480a98f0 legacy=0x7f3a48137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.154+0000 7f3a3dffb640 1 -- 172.21.15.44:0/1920957045 >> v1:172.21.15.111:6789/0 conn(0x7f3a4813f710 legacy=0x7f3a481d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.154+0000 7f3a3dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:42:03.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.154+0000 7f3a3dffb640 1 monclient: found mon.a 2024-03-25T19:42:03.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.154+0000 7f3a3dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:03.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.154+0000 7f3a3dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:42:03.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.154+0000 7f3a3dffb640 1 -- 172.21.15.44:0/1920957045 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a481d84d0 con 0x7f3a481430c0 2024-03-25T19:42:03.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.154+0000 7f3a3dffb640 10 monclient: _finish_auth 0 2024-03-25T19:42:03.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.154+0000 7f3a3dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:42:03.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.154+0000 7f3a3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:03.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.154+0000 7f3a4f614640 5 monclient: authenticate success, global_id 4536 2024-03-25T19:42:03.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.154+0000 7f3a4f614640 10 monclient: _renew_subs 2024-03-25T19:42:03.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.154+0000 7f3a4f614640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:42:03.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.154+0000 7f3a4f614640 1 -- 172.21.15.44:0/1920957045 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a481d41e0 con 0x7f3a481430c0 2024-03-25T19:42:03.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.154+0000 7f3a4f614640 10 client.4536.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:03.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.154+0000 7f3a4f614640 10 monclient: _renew_subs 2024-03-25T19:42:03.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.154+0000 7f3a4f614640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:42:03.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.154+0000 7f3a4f614640 1 -- 172.21.15.44:0/1920957045 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a481d5150 con 0x7f3a481430c0 2024-03-25T19:42:03.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.154+0000 7f3a4f614640 1 librados: init done 2024-03-25T19:42:03.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.155+0000 7f3a4f614640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:03.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.155+0000 7f3a4f614640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:03.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.155+0000 7f3a4f614640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:42:03.157 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.155+0000 7f3a4f614640 1 -- 172.21.15.44:0/1920957045 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a480627b0 con 0x7f3a481430c0 2024-03-25T19:42:03.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.462+0000 7f3a3dffb640 1 -- 172.21.15.44:0/1920957045 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a40001fb0 con 0x7f3a481430c0 2024-03-25T19:42:03.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.462+0000 7f3a3dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:03.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.462+0000 7f3a3dffb640 1 -- 172.21.15.44:0/1920957045 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3a40003b10 con 0x7f3a481430c0 2024-03-25T19:42:03.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.462+0000 7f3a3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:03.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.462+0000 7f3a3dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:03.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.462+0000 7f3a3dffb640 10 monclient: dump: 2024-03-25T19:42:03.463 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:03.463 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:03.463 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:03.464 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:03.464 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:03.464 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:03.464 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:03.464 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:03.464 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:03.464 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:03.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.462+0000 7f3a3f7fe640 4 set_mon_vals no callback set 2024-03-25T19:42:03.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.463+0000 7f3a3dffb640 1 -- 172.21.15.44:0/1920957045 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f3a40003d30 con 0x7f3a481430c0 2024-03-25T19:42:03.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.464+0000 7f3a4d3b0640 10 monclient: get_auth_request con 0x7f3a2c06df70 auth_method 0 2024-03-25T19:42:03.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.464+0000 7f3a3dffb640 10 client.4536.objecter ms_handle_connect 0x7f3a2c06df70 2024-03-25T19:42:03.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.470+0000 7f3a3dffb640 1 -- 172.21.15.44:0/1920957045 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f3a40053780 con 0x7f3a481430c0 2024-03-25T19:42:03.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.470+0000 7f3a3dffb640 10 client.4536.objecter ms_dispatch 0x7f3a48000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:03.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.470+0000 7f3a3dffb640 3 client.4536.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:03.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.470+0000 7f3a3dffb640 3 client.4536.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:03.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.471+0000 7f3a3dffb640 20 client.4536.objecter dump_active .. 0 homeless 2024-03-25T19:42:03.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.471+0000 7f3a3dffb640 1 -- 172.21.15.44:0/1920957045 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3a4004f290 con 0x7f3a481430c0 2024-03-25T19:42:03.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.471+0000 7f3a3dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:03.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.471+0000 7f3a3dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:03.693 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.693+0000 7f3a4f614640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:03.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.693+0000 7f3a4f614640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:03.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.693+0000 7f3a4f614640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:42:03.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.693+0000 7f3a4f614640 1 -- 172.21.15.44:0/1920957045 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a48074cc0 con 0x7f3a481430c0 2024-03-25T19:42:03.721 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.721+0000 7f3a3dffb640 1 -- 172.21.15.44:0/1920957045 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f3a400511c0 con 0x7f3a481430c0 2024-03-25T19:42:03.721 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.721+0000 7f3a3dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:03.721 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.721+0000 7f3a3dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:03.721 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:42:03.721 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:03.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.723+0000 7f3a4f614640 10 librados: watch_flush enter 2024-03-25T19:42:03.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.723+0000 7f3a4f614640 10 librados: watch_flush exit 2024-03-25T19:42:03.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.723+0000 7f3a4f614640 20 client.4536.objecter shutdown clearing up homeless session... 2024-03-25T19:42:03.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.723+0000 7f3a4f614640 10 client.4536.objecter successfully canceled tick 2024-03-25T19:42:03.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.723+0000 7f3a4f614640 1 -- 172.21.15.44:0/1920957045 >> v1:172.21.15.111:6800/3883215793 conn(0x7f3a2c06df70 legacy=0x7f3a2c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.723+0000 7f3a4f614640 10 monclient: shutdown 2024-03-25T19:42:03.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.723+0000 7f3a4f614640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:03.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.723+0000 7f3a4f614640 1 -- 172.21.15.44:0/1920957045 >> v1:172.21.15.44:6789/0 conn(0x7f3a481430c0 legacy=0x7f3a481d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.723+0000 7f3a4f614640 1 -- 172.21.15.44:0/1920957045 shutdown_connections 2024-03-25T19:42:03.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.723+0000 7f3a4f614640 1 -- 172.21.15.44:0/1920957045 >> 172.21.15.44:0/1920957045 conn(0x7f3a480b7ea0 msgr2=0x7f3a48074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:03.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.724+0000 7f3a4f614640 1 -- 172.21.15.44:0/1920957045 shutdown_connections 2024-03-25T19:42:03.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.724+0000 7f3a4f614640 1 -- 172.21.15.44:0/1920957045 wait complete. 2024-03-25T19:42:03.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:03.724+0000 7f3a4f614640 1 librados: shutdown 2024-03-25T19:42:03.736 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:10.542 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:10.657 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.657+0000 7fe34a086640 10 monclient: get_monmap_and_config 2024-03-25T19:42:10.658 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.657+0000 7fe34a086640 10 monclient: build_initial_monmap 2024-03-25T19:42:10.658 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.657+0000 7fe34a086640 10 monclient: monmap: 2024-03-25T19:42:10.658 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:42:10.658 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:10.658 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:42:10.658443+0000 2024-03-25T19:42:10.659 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:42:10.658443+0000 2024-03-25T19:42:10.659 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:10.659 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:10.659 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:42:10.659 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:42:10.660 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:42:10.660 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:10.662 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.658+0000 7fe34a086640 1 Processor -- start 2024-03-25T19:42:10.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.658+0000 7fe34a086640 1 -- start start 2024-03-25T19:42:10.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.658+0000 7fe34a086640 10 monclient: init 2024-03-25T19:42:10.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.658+0000 7fe34a086640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:10.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.658+0000 7fe34a086640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:10.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.658+0000 7fe34a086640 10 monclient(hunting): picked mon.noname-b con 0x7fe3440a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T19:42:10.664 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.658+0000 7fe34a086640 10 monclient(hunting): picked mon.noname-a con 0x7fe34413f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:42:10.664 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.658+0000 7fe34a086640 10 monclient(hunting): picked mon.noname-c con 0x7fe3441430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:42:10.664 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.658+0000 7fe34a086640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe344142520 con 0x7fe34413f710 2024-03-25T19:42:10.664 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.659+0000 7fe34a086640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3440b28a0 con 0x7fe3441430c0 2024-03-25T19:42:10.664 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.659+0000 7fe34a086640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3440b2a30 con 0x7fe3440a98f0 2024-03-25T19:42:10.664 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.659+0000 7fe34a086640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:10.665 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.659+0000 7fe34a086640 10 monclient(hunting): authenticate will time out at 5794.829857s 2024-03-25T19:42:10.665 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.664+0000 7fe3437fe640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fe3440a98f0 0x7fe3440a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:46278/0 (socket says 172.21.15.44:46278) 2024-03-25T19:42:10.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.664+0000 7fe3437fe640 1 -- 172.21.15.44:0/2469000229 learned_addr learned my addr 172.21.15.44:0/2469000229 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:42:10.683 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.683+0000 7fe3427fc640 1 -- 172.21.15.44:0/2469000229 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397537799 0 0) 0x7fe3440b2a30 con 0x7fe3440a98f0 2024-03-25T19:42:10.683 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.683+0000 7fe3427fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:10.683 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.683+0000 7fe3427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:10.683 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.683+0000 7fe3427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:10.683 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.683+0000 7fe3427fc640 10 monclient(hunting): my global_id is 4511 2024-03-25T19:42:10.683 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.683+0000 7fe3427fc640 1 -- 172.21.15.44:0/2469000229 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe32c003520 con 0x7fe3440a98f0 2024-03-25T19:42:10.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.686+0000 7fe3427fc640 1 -- 172.21.15.44:0/2469000229 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643827908 0 0) 0x7fe3440b28a0 con 0x7fe3441430c0 2024-03-25T19:42:10.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.686+0000 7fe3427fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:10.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.686+0000 7fe3427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:10.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.686+0000 7fe3427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:10.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.686+0000 7fe3427fc640 10 monclient(hunting): my global_id is 4522 2024-03-25T19:42:10.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.686+0000 7fe3427fc640 1 -- 172.21.15.44:0/2469000229 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3440b2a30 con 0x7fe3441430c0 2024-03-25T19:42:10.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.691+0000 7fe3427fc640 1 -- 172.21.15.44:0/2469000229 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553125935 0 0) 0x7fe344142520 con 0x7fe34413f710 2024-03-25T19:42:10.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.691+0000 7fe3427fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:10.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.691+0000 7fe3427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:10.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.691+0000 7fe3427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:10.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.691+0000 7fe3427fc640 10 monclient(hunting): my global_id is 4539 2024-03-25T19:42:10.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.691+0000 7fe3427fc640 1 -- 172.21.15.44:0/2469000229 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3440b28a0 con 0x7fe34413f710 2024-03-25T19:42:10.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.699+0000 7fe3427fc640 1 -- 172.21.15.44:0/2469000229 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3600492255 0 0) 0x7fe3440b2a30 con 0x7fe3441430c0 2024-03-25T19:42:10.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.699+0000 7fe3427fc640 1 -- 172.21.15.44:0/2469000229 >> v1:172.21.15.44:6790/0 conn(0x7fe3440a98f0 legacy=0x7fe3440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:10.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.699+0000 7fe3427fc640 1 -- 172.21.15.44:0/2469000229 >> v1:172.21.15.44:6789/0 conn(0x7fe34413f710 legacy=0x7fe3441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:10.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.699+0000 7fe3427fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:10.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.699+0000 7fe3427fc640 1 monclient: found mon.noname-c 2024-03-25T19:42:10.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.699+0000 7fe3427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:10.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.699+0000 7fe3427fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:42:10.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.699+0000 7fe3427fc640 1 -- 172.21.15.44:0/2469000229 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3440b44c0 con 0x7fe3441430c0 2024-03-25T19:42:10.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.699+0000 7fe3427fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:10.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.699+0000 7fe3427fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:10.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.699+0000 7fe3427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:10.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.699+0000 7fe3427fc640 1 -- 172.21.15.44:0/2469000229 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fe33c0019f0 con 0x7fe3441430c0 2024-03-25T19:42:10.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.699+0000 7fe3427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:10.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.699+0000 7fe3427fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:10.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.699+0000 7fe3427fc640 10 monclient: dump: 2024-03-25T19:42:10.700 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:10.700 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:10.700 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:10.700 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:10.701 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:10.701 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:10.701 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:10.701 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:10.701 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:10.701 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:10.701 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.699+0000 7fe34a086640 5 monclient: authenticate success, global_id 4522 2024-03-25T19:42:10.701 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.699+0000 7fe34a086640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:10.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.723+0000 7fe3427fc640 1 -- 172.21.15.44:0/2469000229 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe33c001b50 con 0x7fe3441430c0 2024-03-25T19:42:10.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.723+0000 7fe3427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:10.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.723+0000 7fe3427fc640 1 -- 172.21.15.44:0/2469000229 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fe33c001f60 con 0x7fe3441430c0 2024-03-25T19:42:10.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.723+0000 7fe3427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:10.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.723+0000 7fe3427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:10.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.723+0000 7fe3427fc640 10 monclient: dump: 2024-03-25T19:42:10.723 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:10.723 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:10.723 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:10.723 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:10.723 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:10.723 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:10.723 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:10.724 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:10.724 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:10.724 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:10.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.723+0000 7fe34a086640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:10.724 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.723+0000 7fe34a086640 4 set_mon_vals no callback set 2024-03-25T19:42:10.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.723+0000 7fe34a086640 10 monclient: shutdown 2024-03-25T19:42:10.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.723+0000 7fe34a086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:10.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.723+0000 7fe34a086640 1 -- 172.21.15.44:0/2469000229 >> v1:172.21.15.111:6789/0 conn(0x7fe3441430c0 legacy=0x7fe34414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:10.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.724+0000 7fe34a086640 1 -- 172.21.15.44:0/2469000229 shutdown_connections 2024-03-25T19:42:10.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.724+0000 7fe34a086640 1 -- 172.21.15.44:0/2469000229 >> 172.21.15.44:0/2469000229 conn(0x7fe3440b7ea0 msgr2=0x7fe3440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:10.726 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.724+0000 7fe34a086640 1 -- 172.21.15.44:0/2469000229 shutdown_connections 2024-03-25T19:42:10.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.724+0000 7fe34a086640 1 -- 172.21.15.44:0/2469000229 wait complete. 2024-03-25T19:42:10.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.724+0000 7fe34a086640 10 monclient: build_initial_monmap 2024-03-25T19:42:10.727 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.724+0000 7fe34a086640 10 monclient: monmap: 2024-03-25T19:42:10.727 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:42:10.727 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:10.727 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:42:10.727 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:42:10.727 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:10.727 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:42:10.728 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:42:10.728 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:42:10.728 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:42:10.728 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:10.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.725+0000 7fe34a086640 1 librados: starting msgr at 2024-03-25T19:42:10.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.725+0000 7fe34a086640 1 librados: starting objecter 2024-03-25T19:42:10.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.725+0000 7fe34a086640 1 Processor -- start 2024-03-25T19:42:10.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.725+0000 7fe34a086640 1 -- start start 2024-03-25T19:42:10.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.726+0000 7fe34a086640 1 librados: setting wanted keys 2024-03-25T19:42:10.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.726+0000 7fe34a086640 1 librados: calling monclient init 2024-03-25T19:42:10.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.726+0000 7fe34a086640 10 monclient: init 2024-03-25T19:42:10.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.726+0000 7fe34a086640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:10.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.726+0000 7fe34a086640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:10.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.726+0000 7fe34a086640 10 monclient(hunting): picked mon.noname-a con 0x7fe3440a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:42:10.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.726+0000 7fe34a086640 10 monclient(hunting): picked mon.noname-b con 0x7fe34413f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:42:10.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.726+0000 7fe34a086640 10 monclient(hunting): picked mon.noname-c con 0x7fe3441430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:42:10.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.726+0000 7fe34a086640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3441388a0 con 0x7fe3440a98f0 2024-03-25T19:42:10.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.726+0000 7fe34a086640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3441349e0 con 0x7fe34413f710 2024-03-25T19:42:10.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.726+0000 7fe34a086640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe344138d30 con 0x7fe3441430c0 2024-03-25T19:42:10.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.726+0000 7fe34a086640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:10.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.726+0000 7fe34a086640 10 monclient(hunting): authenticate will time out at 5794.897757s 2024-03-25T19:42:10.734 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.734+0000 7fe343fff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fe3441430c0 0x7fe3441342d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:46328/0 (socket says 172.21.15.44:46328) 2024-03-25T19:42:10.734 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.734+0000 7fe343fff640 1 -- 172.21.15.44:0/2146461210 learned_addr learned my addr 172.21.15.44:0/2146461210 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:42:10.738 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.738+0000 7fe32bfff640 10 client.?.objecter ms_handle_connect 0x7fe3441430c0 2024-03-25T19:42:10.738 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.738+0000 7fe32bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:10.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.743+0000 7fe32bfff640 10 client.?.objecter ms_handle_connect 0x7fe34413f710 2024-03-25T19:42:10.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.743+0000 7fe32bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:10.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.744+0000 7fe32bfff640 10 client.?.objecter ms_handle_connect 0x7fe3440a98f0 2024-03-25T19:42:10.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.744+0000 7fe32bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:10.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.752+0000 7fe32bfff640 1 -- 172.21.15.44:0/2146461210 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263570952 0 0) 0x7fe344138d30 con 0x7fe3441430c0 2024-03-25T19:42:10.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.752+0000 7fe32bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:10.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.752+0000 7fe32bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:10.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.752+0000 7fe32bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:10.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.752+0000 7fe32bfff640 10 monclient(hunting): my global_id is 4514 2024-03-25T19:42:10.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.752+0000 7fe32bfff640 1 -- 172.21.15.44:0/2146461210 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe330004eb0 con 0x7fe3441430c0 2024-03-25T19:42:10.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.752+0000 7fe32bfff640 1 -- 172.21.15.44:0/2146461210 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266881139 0 0) 0x7fe3441388a0 con 0x7fe3440a98f0 2024-03-25T19:42:10.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.752+0000 7fe32bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:10.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.752+0000 7fe32bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:10.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.752+0000 7fe32bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:10.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.752+0000 7fe32bfff640 10 monclient(hunting): my global_id is 4542 2024-03-25T19:42:10.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.752+0000 7fe32bfff640 1 -- 172.21.15.44:0/2146461210 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe344138d30 con 0x7fe3440a98f0 2024-03-25T19:42:10.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.756+0000 7fe32bfff640 1 -- 172.21.15.44:0/2146461210 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 584122 0 0) 0x7fe3441349e0 con 0x7fe34413f710 2024-03-25T19:42:10.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.756+0000 7fe32bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:10.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.756+0000 7fe32bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:10.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.756+0000 7fe32bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:10.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.756+0000 7fe32bfff640 10 monclient(hunting): my global_id is 4525 2024-03-25T19:42:10.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.756+0000 7fe32bfff640 1 -- 172.21.15.44:0/2146461210 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3441388a0 con 0x7fe34413f710 2024-03-25T19:42:10.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.769+0000 7fe32bfff640 1 -- 172.21.15.44:0/2146461210 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3134749182 0 0) 0x7fe330004eb0 con 0x7fe3441430c0 2024-03-25T19:42:10.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.769+0000 7fe32bfff640 1 -- 172.21.15.44:0/2146461210 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3441349e0 con 0x7fe3441430c0 2024-03-25T19:42:10.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.769+0000 7fe32bfff640 1 -- 172.21.15.44:0/2146461210 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fe33c001fa0 con 0x7fe3441430c0 2024-03-25T19:42:10.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.769+0000 7fe32bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:10.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.769+0000 7fe32bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:10.769 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.769+0000 7fe32bfff640 10 monclient(hunting): dump: 2024-03-25T19:42:10.769 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:10.769 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:10.769 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:10.769 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:10.769 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:10.770 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:10.770 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:10.770 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:10.770 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:10.770 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:10.770 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.769+0000 7fe32bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:10.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.770+0000 7fe32bfff640 1 -- 172.21.15.44:0/2146461210 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2564444980 0 0) 0x7fe3441388a0 con 0x7fe34413f710 2024-03-25T19:42:10.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.770+0000 7fe32bfff640 1 -- 172.21.15.44:0/2146461210 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe330004eb0 con 0x7fe34413f710 2024-03-25T19:42:10.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.770+0000 7fe32bfff640 1 -- 172.21.15.44:0/2146461210 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fe338001830 con 0x7fe34413f710 2024-03-25T19:42:10.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.770+0000 7fe32bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:10.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.770+0000 7fe32bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:10.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.770+0000 7fe32bfff640 10 monclient(hunting): dump: 2024-03-25T19:42:10.771 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:10.771 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:10.771 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:10.771 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:10.771 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:10.771 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:10.771 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:10.772 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:10.772 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:10.772 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:10.772 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.772+0000 7fe32bfff640 1 -- 172.21.15.44:0/2146461210 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 180502918 0 0) 0x7fe344138d30 con 0x7fe3440a98f0 2024-03-25T19:42:10.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.772+0000 7fe32bfff640 1 -- 172.21.15.44:0/2146461210 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3441388a0 con 0x7fe3440a98f0 2024-03-25T19:42:10.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.772+0000 7fe32bfff640 1 -- 172.21.15.44:0/2146461210 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fe3340018d0 con 0x7fe3440a98f0 2024-03-25T19:42:10.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.772+0000 7fe32bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:10.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.772+0000 7fe32bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:10.773 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.772+0000 7fe32bfff640 10 monclient(hunting): dump: 2024-03-25T19:42:10.773 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:10.773 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:10.773 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:10.773 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:10.773 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:10.773 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:10.774 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:10.774 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:10.774 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:10.774 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:10.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.779+0000 7fe32bfff640 1 -- 172.21.15.44:0/2146461210 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1344935012 0 0) 0x7fe330004eb0 con 0x7fe34413f710 2024-03-25T19:42:10.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.779+0000 7fe32bfff640 1 -- 172.21.15.44:0/2146461210 >> v1:172.21.15.44:6790/0 conn(0x7fe3441430c0 legacy=0x7fe3441342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:10.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.779+0000 7fe32bfff640 1 -- 172.21.15.44:0/2146461210 >> v1:172.21.15.44:6789/0 conn(0x7fe3440a98f0 legacy=0x7fe3441da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:10.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.779+0000 7fe32bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:10.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.779+0000 7fe32bfff640 1 monclient: found mon.b 2024-03-25T19:42:10.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.779+0000 7fe32bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:10.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.779+0000 7fe32bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:42:10.780 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.779+0000 7fe32bfff640 1 -- 172.21.15.44:0/2146461210 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3440aba50 con 0x7fe34413f710 2024-03-25T19:42:10.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.779+0000 7fe32bfff640 10 monclient: _finish_auth 0 2024-03-25T19:42:10.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.779+0000 7fe32bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:10.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.779+0000 7fe32bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:10.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.779+0000 7fe34a086640 5 monclient: authenticate success, global_id 4525 2024-03-25T19:42:10.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.779+0000 7fe34a086640 10 monclient: _renew_subs 2024-03-25T19:42:10.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.779+0000 7fe34a086640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:42:10.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.779+0000 7fe34a086640 1 -- 172.21.15.44:0/2146461210 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3440b1770 con 0x7fe34413f710 2024-03-25T19:42:10.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.779+0000 7fe34a086640 10 client.4525.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:10.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.779+0000 7fe34a086640 10 monclient: _renew_subs 2024-03-25T19:42:10.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.779+0000 7fe34a086640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:42:10.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.779+0000 7fe34a086640 1 -- 172.21.15.44:0/2146461210 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe344137ba0 con 0x7fe34413f710 2024-03-25T19:42:10.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.779+0000 7fe34a086640 1 librados: init done 2024-03-25T19:42:10.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.780+0000 7fe34a086640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:10.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.780+0000 7fe34a086640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:10.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.780+0000 7fe34a086640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:42:10.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:10.780+0000 7fe34a086640 1 -- 172.21.15.44:0/2146461210 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3440627b0 con 0x7fe34413f710 2024-03-25T19:42:11.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.065+0000 7fe32bfff640 1 -- 172.21.15.44:0/2146461210 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe338001d40 con 0x7fe34413f710 2024-03-25T19:42:11.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.065+0000 7fe32bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:11.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.065+0000 7fe341ffb640 4 set_mon_vals no callback set 2024-03-25T19:42:11.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.065+0000 7fe32bfff640 1 -- 172.21.15.44:0/2146461210 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fe3380037b0 con 0x7fe34413f710 2024-03-25T19:42:11.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.065+0000 7fe32bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:11.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.065+0000 7fe32bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:11.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.065+0000 7fe32bfff640 10 monclient: dump: 2024-03-25T19:42:11.066 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:11.066 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:11.066 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:11.066 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:11.066 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:11.066 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:11.066 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:11.066 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:11.066 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:11.066 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:11.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.066+0000 7fe32bfff640 1 -- 172.21.15.44:0/2146461210 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7fe338019840 con 0x7fe34413f710 2024-03-25T19:42:11.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.067+0000 7fe3437fe640 10 monclient: get_auth_request con 0x7fe33006e160 auth_method 0 2024-03-25T19:42:11.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.067+0000 7fe32bfff640 10 client.4525.objecter ms_handle_connect 0x7fe33006e160 2024-03-25T19:42:11.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.071+0000 7fe32bfff640 1 -- 172.21.15.44:0/2146461210 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7fe33804f040 con 0x7fe34413f710 2024-03-25T19:42:11.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.072+0000 7fe32bfff640 10 client.4525.objecter ms_dispatch 0x7fe344000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:11.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.072+0000 7fe32bfff640 3 client.4525.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:11.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.072+0000 7fe32bfff640 3 client.4525.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:11.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.072+0000 7fe32bfff640 20 client.4525.objecter dump_active .. 0 homeless 2024-03-25T19:42:11.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.072+0000 7fe32bfff640 1 -- 172.21.15.44:0/2146461210 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe338052be0 con 0x7fe34413f710 2024-03-25T19:42:11.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.072+0000 7fe32bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:11.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.072+0000 7fe32bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:11.332 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.332+0000 7fe34a086640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:11.332 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.332+0000 7fe34a086640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:11.332 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.332+0000 7fe34a086640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:42:11.332 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.332+0000 7fe34a086640 1 -- 172.21.15.44:0/2146461210 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe344141a00 con 0x7fe34413f710 2024-03-25T19:42:11.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.352+0000 7fe32bfff640 1 -- 172.21.15.44:0/2146461210 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7fe338050f50 con 0x7fe34413f710 2024-03-25T19:42:11.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.352+0000 7fe32bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:11.352 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.352+0000 7fe32bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:11.353 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:42:11.353 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:11.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.355+0000 7fe34a086640 10 librados: watch_flush enter 2024-03-25T19:42:11.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.355+0000 7fe34a086640 10 librados: watch_flush exit 2024-03-25T19:42:11.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.355+0000 7fe34a086640 20 client.4525.objecter shutdown clearing up homeless session... 2024-03-25T19:42:11.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.355+0000 7fe34a086640 10 client.4525.objecter successfully canceled tick 2024-03-25T19:42:11.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.355+0000 7fe34a086640 1 -- 172.21.15.44:0/2146461210 >> v1:172.21.15.111:6800/3883215793 conn(0x7fe33006e160 legacy=0x7fe330070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:11.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.355+0000 7fe34a086640 10 monclient: shutdown 2024-03-25T19:42:11.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.355+0000 7fe34a086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:11.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.355+0000 7fe34a086640 1 -- 172.21.15.44:0/2146461210 >> v1:172.21.15.111:6789/0 conn(0x7fe34413f710 legacy=0x7fe344133bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:11.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.355+0000 7fe34a086640 1 -- 172.21.15.44:0/2146461210 shutdown_connections 2024-03-25T19:42:11.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.355+0000 7fe34a086640 1 -- 172.21.15.44:0/2146461210 >> 172.21.15.44:0/2146461210 conn(0x7fe3440b7ea0 msgr2=0x7fe344074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:11.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.355+0000 7fe34a086640 1 -- 172.21.15.44:0/2146461210 shutdown_connections 2024-03-25T19:42:11.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.355+0000 7fe34a086640 1 -- 172.21.15.44:0/2146461210 wait complete. 2024-03-25T19:42:11.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:11.355+0000 7fe34a086640 1 librados: shutdown 2024-03-25T19:42:11.367 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:18.174 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:18.287 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.287+0000 7effe28f6640 10 monclient: get_monmap_and_config 2024-03-25T19:42:18.287 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.287+0000 7effe28f6640 10 monclient: build_initial_monmap 2024-03-25T19:42:18.287 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.287+0000 7effe28f6640 10 monclient: monmap: 2024-03-25T19:42:18.287 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:42:18.287 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:18.287 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:42:18.288572+0000 2024-03-25T19:42:18.287 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:42:18.288572+0000 2024-03-25T19:42:18.287 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:18.288 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:18.288 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:42:18.288 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:42:18.288 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:42:18.288 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:18.288 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.288+0000 7effe28f6640 1 Processor -- start 2024-03-25T19:42:18.288 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.288+0000 7effe28f6640 1 -- start start 2024-03-25T19:42:18.288 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.288+0000 7effe28f6640 10 monclient: init 2024-03-25T19:42:18.289 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.289+0000 7effe28f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:18.289 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.289+0000 7effe28f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:18.289 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.289+0000 7effe28f6640 10 monclient(hunting): picked mon.noname-a con 0x7effdc0a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:42:18.289 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.289+0000 7effe28f6640 10 monclient(hunting): picked mon.noname-c con 0x7effdc13f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:42:18.289 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.289+0000 7effe28f6640 10 monclient(hunting): picked mon.noname-b con 0x7effdc1430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:42:18.289 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.289+0000 7effe28f6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effdc142520 con 0x7effdc0a98f0 2024-03-25T19:42:18.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.289+0000 7effe28f6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effdc0b28a0 con 0x7effdc13f710 2024-03-25T19:42:18.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.289+0000 7effe28f6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effdc0b2a30 con 0x7effdc1430c0 2024-03-25T19:42:18.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.289+0000 7effe28f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:18.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.289+0000 7effe28f6640 10 monclient(hunting): authenticate will time out at 5802.460328s 2024-03-25T19:42:18.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.298+0000 7effe0e93640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7effdc1430c0 0x7effdc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:50950/0 (socket says 172.21.15.44:50950) 2024-03-25T19:42:18.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.298+0000 7effe0e93640 1 -- 172.21.15.44:0/142329677 learned_addr learned my addr 172.21.15.44:0/142329677 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:42:18.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.320+0000 7effdaffd640 1 -- 172.21.15.44:0/142329677 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319913178 0 0) 0x7effdc0b2a30 con 0x7effdc1430c0 2024-03-25T19:42:18.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.320+0000 7effdaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:18.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.320+0000 7effdaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:18.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.320+0000 7effdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:18.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.320+0000 7effdaffd640 10 monclient(hunting): my global_id is 4517 2024-03-25T19:42:18.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.320+0000 7effdaffd640 1 -- 172.21.15.44:0/142329677 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effc4003520 con 0x7effdc1430c0 2024-03-25T19:42:18.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.329+0000 7effdaffd640 1 -- 172.21.15.44:0/142329677 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927107452 0 0) 0x7effdc0b28a0 con 0x7effdc13f710 2024-03-25T19:42:18.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.329+0000 7effdaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:18.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.329+0000 7effdaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:18.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.329+0000 7effdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:18.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.329+0000 7effdaffd640 10 monclient(hunting): my global_id is 4528 2024-03-25T19:42:18.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.329+0000 7effdaffd640 1 -- 172.21.15.44:0/142329677 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effdc0b2a30 con 0x7effdc13f710 2024-03-25T19:42:18.333 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.333+0000 7effdaffd640 1 -- 172.21.15.44:0/142329677 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3682204226 0 0) 0x7effc4003520 con 0x7effdc1430c0 2024-03-25T19:42:18.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.333+0000 7effdaffd640 1 -- 172.21.15.44:0/142329677 >> v1:172.21.15.111:6789/0 conn(0x7effdc13f710 legacy=0x7effdc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:18.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.333+0000 7effdaffd640 1 -- 172.21.15.44:0/142329677 >> v1:172.21.15.44:6789/0 conn(0x7effdc0a98f0 legacy=0x7effdc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:18.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.333+0000 7effdaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:18.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.333+0000 7effdaffd640 1 monclient: found mon.noname-b 2024-03-25T19:42:18.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.333+0000 7effdaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:18.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.333+0000 7effdaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:42:18.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.333+0000 7effdaffd640 1 -- 172.21.15.44:0/142329677 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effdc0b44c0 con 0x7effdc1430c0 2024-03-25T19:42:18.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.333+0000 7effdaffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:18.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.333+0000 7effdaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:18.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.333+0000 7effdaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:18.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.333+0000 7effdaffd640 1 -- 172.21.15.44:0/142329677 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7effd4001a10 con 0x7effdc1430c0 2024-03-25T19:42:18.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.333+0000 7effdaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:18.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.333+0000 7effdaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:18.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.333+0000 7effdaffd640 10 monclient: dump: 2024-03-25T19:42:18.335 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:18.335 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:18.335 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:18.335 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:18.335 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:18.335 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:18.335 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:18.335 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:18.335 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:18.335 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:18.335 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.333+0000 7effe28f6640 5 monclient: authenticate success, global_id 4517 2024-03-25T19:42:18.335 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.333+0000 7effe28f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:18.337 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.337+0000 7effdaffd640 1 -- 172.21.15.44:0/142329677 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7effd4001b70 con 0x7effdc1430c0 2024-03-25T19:42:18.337 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.337+0000 7effdaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:18.337 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.337+0000 7effe28f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:18.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.337+0000 7effe28f6640 4 set_mon_vals no callback set 2024-03-25T19:42:18.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.338+0000 7effe28f6640 10 monclient: shutdown 2024-03-25T19:42:18.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.338+0000 7effe28f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:18.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.338+0000 7effe28f6640 1 -- 172.21.15.44:0/142329677 >> v1:172.21.15.44:6790/0 conn(0x7effdc1430c0 legacy=0x7effdc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:18.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.338+0000 7effe28f6640 1 -- 172.21.15.44:0/142329677 shutdown_connections 2024-03-25T19:42:18.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.338+0000 7effe28f6640 1 -- 172.21.15.44:0/142329677 >> 172.21.15.44:0/142329677 conn(0x7effdc0b7ea0 msgr2=0x7effdc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:18.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.338+0000 7effe28f6640 1 -- 172.21.15.44:0/142329677 shutdown_connections 2024-03-25T19:42:18.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.338+0000 7effe28f6640 1 -- 172.21.15.44:0/142329677 wait complete. 2024-03-25T19:42:18.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.339+0000 7effe28f6640 10 monclient: build_initial_monmap 2024-03-25T19:42:18.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.339+0000 7effe28f6640 10 monclient: monmap: 2024-03-25T19:42:18.340 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:42:18.340 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:18.340 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:42:18.340 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:42:18.340 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:18.340 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:42:18.340 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:42:18.340 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:42:18.340 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:42:18.340 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:18.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.339+0000 7effe28f6640 1 librados: starting msgr at 2024-03-25T19:42:18.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.339+0000 7effe28f6640 1 librados: starting objecter 2024-03-25T19:42:18.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.339+0000 7effe28f6640 1 Processor -- start 2024-03-25T19:42:18.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.340+0000 7effe28f6640 1 -- start start 2024-03-25T19:42:18.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.340+0000 7effe28f6640 1 librados: setting wanted keys 2024-03-25T19:42:18.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.340+0000 7effe28f6640 1 librados: calling monclient init 2024-03-25T19:42:18.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.340+0000 7effe28f6640 10 monclient: init 2024-03-25T19:42:18.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.340+0000 7effe28f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:18.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.340+0000 7effe28f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:18.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.340+0000 7effe28f6640 10 monclient(hunting): picked mon.noname-b con 0x7effdc0a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:42:18.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.340+0000 7effe28f6640 10 monclient(hunting): picked mon.noname-a con 0x7effdc13f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:42:18.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.340+0000 7effe28f6640 10 monclient(hunting): picked mon.noname-c con 0x7effdc1430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:42:18.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.340+0000 7effe28f6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effdc1d51f0 con 0x7effdc13f710 2024-03-25T19:42:18.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.340+0000 7effe28f6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effdc0abd30 con 0x7effdc0a98f0 2024-03-25T19:42:18.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.340+0000 7effe28f6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effdc138a30 con 0x7effdc1430c0 2024-03-25T19:42:18.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.340+0000 7effe28f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:18.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.340+0000 7effe28f6640 10 monclient(hunting): authenticate will time out at 5802.511579s 2024-03-25T19:42:18.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.355+0000 7effdb7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7effdc13f710 0x7effdc1dad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40436/0 (socket says 172.21.15.44:40436) 2024-03-25T19:42:18.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.355+0000 7effdb7fe640 1 -- 172.21.15.44:0/4013701711 learned_addr learned my addr 172.21.15.44:0/4013701711 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:42:18.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.357+0000 7effd8ff9640 10 client.?.objecter ms_handle_connect 0x7effdc13f710 2024-03-25T19:42:18.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.357+0000 7effd8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:18.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.359+0000 7effd8ff9640 10 client.?.objecter ms_handle_connect 0x7effdc0a98f0 2024-03-25T19:42:18.359 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.359+0000 7effd8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:18.363 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.363+0000 7effd8ff9640 10 client.?.objecter ms_handle_connect 0x7effdc1430c0 2024-03-25T19:42:18.363 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.363+0000 7effd8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:18.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.367+0000 7effd8ff9640 1 -- 172.21.15.44:0/4013701711 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3236967869 0 0) 0x7effdc0abd30 con 0x7effdc0a98f0 2024-03-25T19:42:18.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.367+0000 7effd8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:18.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.367+0000 7effd8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:18.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.367+0000 7effd8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:18.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.367+0000 7effd8ff9640 10 monclient(hunting): my global_id is 4531 2024-03-25T19:42:18.367 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.367+0000 7effd8ff9640 1 -- 172.21.15.44:0/4013701711 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effc8004eb0 con 0x7effdc0a98f0 2024-03-25T19:42:18.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.369+0000 7effd8ff9640 1 -- 172.21.15.44:0/4013701711 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634003304 0 0) 0x7effdc1d51f0 con 0x7effdc13f710 2024-03-25T19:42:18.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.369+0000 7effd8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:18.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.369+0000 7effd8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:18.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.369+0000 7effd8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:18.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.369+0000 7effd8ff9640 10 monclient(hunting): my global_id is 4548 2024-03-25T19:42:18.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.369+0000 7effd8ff9640 1 -- 172.21.15.44:0/4013701711 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effdc0abd30 con 0x7effdc13f710 2024-03-25T19:42:18.385 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.384+0000 7effd8ff9640 1 -- 172.21.15.44:0/4013701711 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3102233128 0 0) 0x7effc8004eb0 con 0x7effdc0a98f0 2024-03-25T19:42:18.385 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.384+0000 7effd8ff9640 1 -- 172.21.15.44:0/4013701711 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effdc1d51f0 con 0x7effdc0a98f0 2024-03-25T19:42:18.385 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.384+0000 7effd8ff9640 1 -- 172.21.15.44:0/4013701711 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7effcc001930 con 0x7effdc0a98f0 2024-03-25T19:42:18.385 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.384+0000 7effd8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:18.385 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.384+0000 7effd8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:18.385 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.384+0000 7effd8ff9640 10 monclient(hunting): dump: 2024-03-25T19:42:18.385 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:18.385 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:18.385 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:18.385 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:18.385 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:18.385 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:18.385 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:18.386 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:18.386 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:18.386 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:18.386 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.384+0000 7effd8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:18.386 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.384+0000 7effd8ff9640 1 -- 172.21.15.44:0/4013701711 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 770112932 0 0) 0x7effdc138a30 con 0x7effdc1430c0 2024-03-25T19:42:18.386 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.384+0000 7effd8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:18.386 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.384+0000 7effd8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:18.386 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.384+0000 7effd8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:18.386 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.384+0000 7effd8ff9640 10 monclient(hunting): my global_id is 4520 2024-03-25T19:42:18.386 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.384+0000 7effd8ff9640 1 -- 172.21.15.44:0/4013701711 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effc8004eb0 con 0x7effdc1430c0 2024-03-25T19:42:18.387 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.386+0000 7effd8ff9640 1 -- 172.21.15.44:0/4013701711 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3426067491 0 0) 0x7effdc0abd30 con 0x7effdc13f710 2024-03-25T19:42:18.387 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.386+0000 7effd8ff9640 1 -- 172.21.15.44:0/4013701711 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effdc138a30 con 0x7effdc13f710 2024-03-25T19:42:18.387 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.386+0000 7effd8ff9640 1 -- 172.21.15.44:0/4013701711 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7effd00019f0 con 0x7effdc13f710 2024-03-25T19:42:18.387 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.386+0000 7effd8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:18.387 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.386+0000 7effd8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:18.387 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.386+0000 7effd8ff9640 10 monclient(hunting): dump: 2024-03-25T19:42:18.387 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:18.387 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:18.387 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:18.388 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:18.388 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:18.388 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:18.388 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:18.388 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:18.388 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:18.388 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:18.390 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.390+0000 7effd8ff9640 1 -- 172.21.15.44:0/4013701711 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 47953799 0 0) 0x7effdc1d51f0 con 0x7effdc0a98f0 2024-03-25T19:42:18.390 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.390+0000 7effd8ff9640 1 -- 172.21.15.44:0/4013701711 >> v1:172.21.15.44:6790/0 conn(0x7effdc1430c0 legacy=0x7effdc1db410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:18.390 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.390+0000 7effd8ff9640 1 -- 172.21.15.44:0/4013701711 >> v1:172.21.15.44:6789/0 conn(0x7effdc13f710 legacy=0x7effdc1dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:18.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.390+0000 7effd8ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:42:18.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.390+0000 7effd8ff9640 1 monclient: found mon.b 2024-03-25T19:42:18.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.390+0000 7effd8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:18.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.390+0000 7effd8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:42:18.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.390+0000 7effd8ff9640 1 -- 172.21.15.44:0/4013701711 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effdc138bc0 con 0x7effdc0a98f0 2024-03-25T19:42:18.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.390+0000 7effd8ff9640 10 monclient: _finish_auth 0 2024-03-25T19:42:18.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.390+0000 7effd8ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:42:18.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.390+0000 7effd8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:18.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.390+0000 7effe28f6640 5 monclient: authenticate success, global_id 4531 2024-03-25T19:42:18.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.390+0000 7effe28f6640 10 monclient: _renew_subs 2024-03-25T19:42:18.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.390+0000 7effe28f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:42:18.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.390+0000 7effe28f6640 1 -- 172.21.15.44:0/4013701711 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effdc1e2dd0 con 0x7effdc0a98f0 2024-03-25T19:42:18.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.390+0000 7effe28f6640 10 client.4531.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:18.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.390+0000 7effe28f6640 10 monclient: _renew_subs 2024-03-25T19:42:18.392 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.390+0000 7effe28f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:42:18.392 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.390+0000 7effe28f6640 1 -- 172.21.15.44:0/4013701711 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7effdc1e3140 con 0x7effdc0a98f0 2024-03-25T19:42:18.392 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.390+0000 7effe28f6640 1 librados: init done 2024-03-25T19:42:18.392 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.391+0000 7effe28f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:18.392 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.391+0000 7effe28f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:18.392 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.391+0000 7effe28f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:42:18.392 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.391+0000 7effe28f6640 1 -- 172.21.15.44:0/4013701711 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effdc0627b0 con 0x7effdc0a98f0 2024-03-25T19:42:18.668 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.668+0000 7effd8ff9640 1 -- 172.21.15.44:0/4013701711 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7effcc0035f0 con 0x7effdc0a98f0 2024-03-25T19:42:18.668 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.668+0000 7effd8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:18.668 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.668+0000 7effd8ff9640 1 -- 172.21.15.44:0/4013701711 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7effcc003900 con 0x7effdc0a98f0 2024-03-25T19:42:18.668 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.668+0000 7effd8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:18.668 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.668+0000 7effd8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:18.668 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.668+0000 7effd8ff9640 10 monclient: dump: 2024-03-25T19:42:18.668 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:18.668 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:18.668 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:18.668 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:18.668 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:18.668 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:18.669 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:18.669 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:18.669 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:18.669 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:18.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.668+0000 7effda7fc640 4 set_mon_vals no callback set 2024-03-25T19:42:18.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.669+0000 7effd8ff9640 1 -- 172.21.15.44:0/4013701711 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7effcc0198d0 con 0x7effdc0a98f0 2024-03-25T19:42:18.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.669+0000 7effdb7fe640 10 monclient: get_auth_request con 0x7effc806df70 auth_method 0 2024-03-25T19:42:18.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.670+0000 7effd8ff9640 10 client.4531.objecter ms_handle_connect 0x7effc806df70 2024-03-25T19:42:18.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.685+0000 7effd8ff9640 1 -- 172.21.15.44:0/4013701711 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7effcc0536b0 con 0x7effdc0a98f0 2024-03-25T19:42:18.759 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.685+0000 7effd8ff9640 10 client.4531.objecter ms_dispatch 0x7effdc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:18.760 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.685+0000 7effd8ff9640 3 client.4531.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:18.760 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.685+0000 7effd8ff9640 3 client.4531.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:18.760 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.685+0000 7effd8ff9640 20 client.4531.objecter dump_active .. 0 homeless 2024-03-25T19:42:18.760 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.686+0000 7effd8ff9640 1 -- 172.21.15.44:0/4013701711 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7effcc04eff0 con 0x7effdc0a98f0 2024-03-25T19:42:18.761 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.686+0000 7effd8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:18.761 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.686+0000 7effd8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:18.950 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.950+0000 7effe28f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:18.950 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.950+0000 7effe28f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:18.950 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.950+0000 7effe28f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:42:18.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.950+0000 7effe28f6640 1 -- 172.21.15.44:0/4013701711 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7effdc14f240 con 0x7effdc0a98f0 2024-03-25T19:42:18.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.989+0000 7effd8ff9640 1 -- 172.21.15.44:0/4013701711 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7effcc0510f0 con 0x7effdc0a98f0 2024-03-25T19:42:18.990 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.989+0000 7effd8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:18.990 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.989+0000 7effd8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:18.990 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:42:18.990 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:18.993 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.992+0000 7effe28f6640 10 librados: watch_flush enter 2024-03-25T19:42:18.993 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.992+0000 7effe28f6640 10 librados: watch_flush exit 2024-03-25T19:42:18.993 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.992+0000 7effe28f6640 20 client.4531.objecter shutdown clearing up homeless session... 2024-03-25T19:42:18.993 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.992+0000 7effe28f6640 10 client.4531.objecter successfully canceled tick 2024-03-25T19:42:18.993 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.992+0000 7effe28f6640 1 -- 172.21.15.44:0/4013701711 >> v1:172.21.15.111:6800/3883215793 conn(0x7effc806df70 legacy=0x7effc8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:18.993 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.992+0000 7effe28f6640 10 monclient: shutdown 2024-03-25T19:42:18.993 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.992+0000 7effe28f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:18.993 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.992+0000 7effe28f6640 1 -- 172.21.15.44:0/4013701711 >> v1:172.21.15.111:6789/0 conn(0x7effdc0a98f0 legacy=0x7effdc138320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:18.993 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.992+0000 7effe28f6640 1 -- 172.21.15.44:0/4013701711 shutdown_connections 2024-03-25T19:42:18.993 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.992+0000 7effe28f6640 1 -- 172.21.15.44:0/4013701711 >> 172.21.15.44:0/4013701711 conn(0x7effdc0b7ea0 msgr2=0x7effdc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:18.993 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.992+0000 7effe28f6640 1 -- 172.21.15.44:0/4013701711 shutdown_connections 2024-03-25T19:42:18.993 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.992+0000 7effe28f6640 1 -- 172.21.15.44:0/4013701711 wait complete. 2024-03-25T19:42:18.993 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:18.993+0000 7effe28f6640 1 librados: shutdown 2024-03-25T19:42:19.006 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:19.988 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:07:10.890 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:19.996 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:07:10.897 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:20.033 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:07:10.934 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:20.048 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:07:10.950 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:20.048 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:07:10.928 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:20.053 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:07:10.955 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:20.056 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:07:10.935 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:20.075 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:07:10.955 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:20.087 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:07:10.988 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:25.813 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:25.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.932+0000 7ffaf424e640 10 monclient: get_monmap_and_config 2024-03-25T19:42:25.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.932+0000 7ffaf424e640 10 monclient: build_initial_monmap 2024-03-25T19:42:25.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.932+0000 7ffaf424e640 10 monclient: monmap: 2024-03-25T19:42:25.932 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:42:25.932 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:25.932 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:42:25.933411+0000 2024-03-25T19:42:25.932 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:42:25.933411+0000 2024-03-25T19:42:25.932 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:25.932 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:25.933 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:42:25.933 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:42:25.933 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:42:25.933 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:25.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.932+0000 7ffaf424e640 1 Processor -- start 2024-03-25T19:42:25.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.933+0000 7ffaf424e640 1 -- start start 2024-03-25T19:42:25.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.933+0000 7ffaf424e640 10 monclient: init 2024-03-25T19:42:25.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.933+0000 7ffaf424e640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:25.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.933+0000 7ffaf424e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:25.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.933+0000 7ffaf424e640 10 monclient(hunting): picked mon.noname-b con 0x7ffaec0a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T19:42:25.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.933+0000 7ffaf424e640 10 monclient(hunting): picked mon.noname-a con 0x7ffaec13f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:42:25.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.933+0000 7ffaf424e640 10 monclient(hunting): picked mon.noname-c con 0x7ffaec1430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:42:25.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.933+0000 7ffaf424e640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec142520 con 0x7ffaec13f710 2024-03-25T19:42:25.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.933+0000 7ffaf424e640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec0b28a0 con 0x7ffaec1430c0 2024-03-25T19:42:25.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.933+0000 7ffaf424e640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec0b2a30 con 0x7ffaec0a98f0 2024-03-25T19:42:25.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.934+0000 7ffaf424e640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:25.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.934+0000 7ffaf424e640 10 monclient(hunting): authenticate will time out at 5810.104855s 2024-03-25T19:42:25.944 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.944+0000 7ffaf27eb640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ffaec1430c0 0x7ffaec14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:48560/0 (socket says 172.21.15.44:48560) 2024-03-25T19:42:25.944 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.944+0000 7ffaf27eb640 1 -- 172.21.15.44:0/830617759 learned_addr learned my addr 172.21.15.44:0/830617759 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:42:25.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.951+0000 7ffaf0fe8640 1 -- 172.21.15.44:0/830617759 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4220061971 0 0) 0x7ffaec0b28a0 con 0x7ffaec1430c0 2024-03-25T19:42:25.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.951+0000 7ffaf0fe8640 10 monclient(hunting): _negotiate 2024-03-25T19:42:25.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.951+0000 7ffaf0fe8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:25.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.951+0000 7ffaf0fe8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:25.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.951+0000 7ffaf0fe8640 10 monclient(hunting): my global_id is 4534 2024-03-25T19:42:25.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.951+0000 7ffaf0fe8640 1 -- 172.21.15.44:0/830617759 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad8003520 con 0x7ffaec1430c0 2024-03-25T19:42:25.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.955+0000 7ffaf0fe8640 1 -- 172.21.15.44:0/830617759 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2009077850 0 0) 0x7ffad8003520 con 0x7ffaec1430c0 2024-03-25T19:42:25.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.955+0000 7ffaf0fe8640 1 -- 172.21.15.44:0/830617759 >> v1:172.21.15.44:6790/0 conn(0x7ffaec0a98f0 legacy=0x7ffaec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:25.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.955+0000 7ffaf0fe8640 1 -- 172.21.15.44:0/830617759 >> v1:172.21.15.44:6789/0 conn(0x7ffaec13f710 legacy=0x7ffaec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:42:25.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.955+0000 7ffaf0fe8640 10 monclient: _finish_hunting 0 2024-03-25T19:42:25.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.955+0000 7ffaf0fe8640 1 monclient: found mon.noname-c 2024-03-25T19:42:25.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.955+0000 7ffaf0fe8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:25.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.955+0000 7ffaf0fe8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:42:25.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.955+0000 7ffaf0fe8640 1 -- 172.21.15.44:0/830617759 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec0b44c0 con 0x7ffaec1430c0 2024-03-25T19:42:25.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.955+0000 7ffaf0fe8640 10 monclient: _finish_auth 0 2024-03-25T19:42:25.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.955+0000 7ffaf0fe8640 10 monclient: _check_auth_tickets 2024-03-25T19:42:25.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.955+0000 7ffaf0fe8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:25.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.955+0000 7ffaf0fe8640 1 -- 172.21.15.44:0/830617759 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ffae4001a10 con 0x7ffaec1430c0 2024-03-25T19:42:25.957 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.955+0000 7ffaf0fe8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:25.957 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.955+0000 7ffaf0fe8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:25.957 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.955+0000 7ffaf0fe8640 10 monclient: dump: 2024-03-25T19:42:25.957 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:25.957 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:25.957 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:25.957 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:25.957 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:25.957 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:25.957 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:25.957 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:25.957 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:25.957 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:25.957 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.955+0000 7ffaf424e640 5 monclient: authenticate success, global_id 4534 2024-03-25T19:42:25.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.955+0000 7ffaf424e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:25.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.958+0000 7ffaf0fe8640 1 -- 172.21.15.44:0/830617759 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffae4001be0 con 0x7ffaec1430c0 2024-03-25T19:42:25.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.958+0000 7ffaf0fe8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:25.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.958+0000 7ffaf424e640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:25.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.958+0000 7ffaf0fe8640 1 -- 172.21.15.44:0/830617759 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ffae4002030 con 0x7ffaec1430c0 2024-03-25T19:42:25.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.958+0000 7ffaf424e640 4 set_mon_vals no callback set 2024-03-25T19:42:25.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.958+0000 7ffaf0fe8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:25.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.958+0000 7ffaf0fe8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:25.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.958+0000 7ffaf0fe8640 10 monclient: dump: 2024-03-25T19:42:25.959 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:25.959 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:25.959 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:25.959 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:25.959 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:25.959 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:25.959 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:25.959 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:25.959 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:25.959 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:25.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.959+0000 7ffaf424e640 10 monclient: shutdown 2024-03-25T19:42:25.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.959+0000 7ffaf424e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:25.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.959+0000 7ffaf424e640 1 -- 172.21.15.44:0/830617759 >> v1:172.21.15.111:6789/0 conn(0x7ffaec1430c0 legacy=0x7ffaec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:25.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.959+0000 7ffaf424e640 1 -- 172.21.15.44:0/830617759 shutdown_connections 2024-03-25T19:42:25.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.959+0000 7ffaf424e640 1 -- 172.21.15.44:0/830617759 >> 172.21.15.44:0/830617759 conn(0x7ffaec0b7ea0 msgr2=0x7ffaec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:25.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.959+0000 7ffaf424e640 1 -- 172.21.15.44:0/830617759 shutdown_connections 2024-03-25T19:42:25.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.959+0000 7ffaf424e640 1 -- 172.21.15.44:0/830617759 wait complete. 2024-03-25T19:42:25.962 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.960+0000 7ffaf424e640 10 monclient: build_initial_monmap 2024-03-25T19:42:25.962 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.960+0000 7ffaf424e640 10 monclient: monmap: 2024-03-25T19:42:25.962 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:42:25.962 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:25.962 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:42:25.962 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:42:25.962 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:25.962 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:42:25.962 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:42:25.962 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:42:25.962 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:42:25.962 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:25.962 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.960+0000 7ffaf424e640 1 librados: starting msgr at 2024-03-25T19:42:25.963 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.960+0000 7ffaf424e640 1 librados: starting objecter 2024-03-25T19:42:25.963 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.960+0000 7ffaf424e640 1 Processor -- start 2024-03-25T19:42:25.963 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.960+0000 7ffaf424e640 1 -- start start 2024-03-25T19:42:25.963 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.960+0000 7ffaf424e640 1 librados: setting wanted keys 2024-03-25T19:42:25.963 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.960+0000 7ffaf424e640 1 librados: calling monclient init 2024-03-25T19:42:25.963 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.960+0000 7ffaf424e640 10 monclient: init 2024-03-25T19:42:25.963 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.961+0000 7ffaf424e640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:25.963 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.961+0000 7ffaf424e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:25.963 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.961+0000 7ffaf424e640 10 monclient(hunting): picked mon.noname-b con 0x7ffaec0a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:42:25.963 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.961+0000 7ffaf424e640 10 monclient(hunting): picked mon.noname-a con 0x7ffaec13f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:42:25.963 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.961+0000 7ffaf424e640 10 monclient(hunting): picked mon.noname-c con 0x7ffaec1430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:42:25.963 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.961+0000 7ffaf424e640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec13b1f0 con 0x7ffaec13f710 2024-03-25T19:42:25.963 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.961+0000 7ffaf424e640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec13b4d0 con 0x7ffaec0a98f0 2024-03-25T19:42:25.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.961+0000 7ffaf424e640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec13b7b0 con 0x7ffaec1430c0 2024-03-25T19:42:25.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.961+0000 7ffaf424e640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:25.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.961+0000 7ffaf424e640 10 monclient(hunting): authenticate will time out at 5810.132084s 2024-03-25T19:42:25.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.972+0000 7ffaf17e9640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7ffaec13f710 0x7ffaec13bbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40706/0 (socket says 172.21.15.44:40706) 2024-03-25T19:42:25.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.972+0000 7ffaf17e9640 1 -- 172.21.15.44:0/1405541239 learned_addr learned my addr 172.21.15.44:0/1405541239 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:42:25.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.973+0000 7ffae27fc640 10 client.?.objecter ms_handle_connect 0x7ffaec13f710 2024-03-25T19:42:25.973 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.973+0000 7ffae27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:25.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.980+0000 7ffae27fc640 1 -- 172.21.15.44:0/1405541239 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1962247780 0 0) 0x7ffaec13b1f0 con 0x7ffaec13f710 2024-03-25T19:42:25.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.980+0000 7ffae27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:25.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.980+0000 7ffae27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:25.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.980+0000 7ffae27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:25.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.980+0000 7ffae27fc640 10 monclient(hunting): my global_id is 4551 2024-03-25T19:42:25.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.980+0000 7ffae27fc640 1 -- 172.21.15.44:0/1405541239 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad0004eb0 con 0x7ffaec13f710 2024-03-25T19:42:25.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.981+0000 7ffae27fc640 10 client.?.objecter ms_handle_connect 0x7ffaec0a98f0 2024-03-25T19:42:25.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.981+0000 7ffae27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:25.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.983+0000 7ffae27fc640 10 client.?.objecter ms_handle_connect 0x7ffaec1430c0 2024-03-25T19:42:25.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.983+0000 7ffae27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:25.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.995+0000 7ffae27fc640 1 -- 172.21.15.44:0/1405541239 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560086129 0 0) 0x7ffaec13b7b0 con 0x7ffaec1430c0 2024-03-25T19:42:25.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.995+0000 7ffae27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:25.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.995+0000 7ffae27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:25.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.995+0000 7ffae27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:25.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.995+0000 7ffae27fc640 10 monclient(hunting): my global_id is 4526 2024-03-25T19:42:25.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:25.995+0000 7ffae27fc640 1 -- 172.21.15.44:0/1405541239 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaec13b1f0 con 0x7ffaec1430c0 2024-03-25T19:42:26.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.000+0000 7ffae27fc640 1 -- 172.21.15.44:0/1405541239 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1457539806 0 0) 0x7ffaec13b4d0 con 0x7ffaec0a98f0 2024-03-25T19:42:26.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.000+0000 7ffae27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:26.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.000+0000 7ffae27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:26.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.000+0000 7ffae27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:26.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.000+0000 7ffae27fc640 10 monclient(hunting): my global_id is 4537 2024-03-25T19:42:26.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.000+0000 7ffae27fc640 1 -- 172.21.15.44:0/1405541239 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaec13b7b0 con 0x7ffaec0a98f0 2024-03-25T19:42:26.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.011+0000 7ffae27fc640 1 -- 172.21.15.44:0/1405541239 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 103542626 0 0) 0x7ffad0004eb0 con 0x7ffaec13f710 2024-03-25T19:42:26.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.011+0000 7ffae27fc640 1 -- 172.21.15.44:0/1405541239 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffaec13b4d0 con 0x7ffaec13f710 2024-03-25T19:42:26.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.011+0000 7ffae27fc640 1 -- 172.21.15.44:0/1405541239 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ffadc004080 con 0x7ffaec13f710 2024-03-25T19:42:26.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.011+0000 7ffae27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:26.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.011+0000 7ffae27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:26.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.011+0000 7ffae27fc640 10 monclient(hunting): dump: 2024-03-25T19:42:26.012 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:26.012 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:26.012 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:26.012 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:26.012 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:26.012 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:26.012 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:26.012 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:26.012 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:26.012 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:26.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.011+0000 7ffae27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:26.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.016+0000 7ffae27fc640 1 -- 172.21.15.44:0/1405541239 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 783504534 0 0) 0x7ffaec13b1f0 con 0x7ffaec1430c0 2024-03-25T19:42:26.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.016+0000 7ffae27fc640 1 -- 172.21.15.44:0/1405541239 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffad0004eb0 con 0x7ffaec1430c0 2024-03-25T19:42:26.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.016+0000 7ffae27fc640 1 -- 172.21.15.44:0/1405541239 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ffae4001ef0 con 0x7ffaec1430c0 2024-03-25T19:42:26.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.016+0000 7ffae27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:26.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.016+0000 7ffae27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:26.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.016+0000 7ffae27fc640 10 monclient(hunting): dump: 2024-03-25T19:42:26.016 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:26.016 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:26.016 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:26.016 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:26.016 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:26.017 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:26.017 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:26.017 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:26.017 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:26.017 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:26.035 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.035+0000 7ffae27fc640 1 -- 172.21.15.44:0/1405541239 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3788975412 0 0) 0x7ffaec13b7b0 con 0x7ffaec0a98f0 2024-03-25T19:42:26.035 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.035+0000 7ffae27fc640 1 -- 172.21.15.44:0/1405541239 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffaec13b1f0 con 0x7ffaec0a98f0 2024-03-25T19:42:26.035 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.035+0000 7ffae27fc640 1 -- 172.21.15.44:0/1405541239 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ffae8001830 con 0x7ffaec0a98f0 2024-03-25T19:42:26.035 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.035+0000 7ffae27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:26.036 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.035+0000 7ffae27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:26.036 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.035+0000 7ffae27fc640 10 monclient(hunting): dump: 2024-03-25T19:42:26.036 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:26.036 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:26.036 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:26.036 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:26.036 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:26.036 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:26.036 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:26.036 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:26.036 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:26.036 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:26.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.039+0000 7ffae27fc640 1 -- 172.21.15.44:0/1405541239 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3106129250 0 0) 0x7ffad0004eb0 con 0x7ffaec1430c0 2024-03-25T19:42:26.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.039+0000 7ffae27fc640 1 -- 172.21.15.44:0/1405541239 >> v1:172.21.15.111:6789/0 conn(0x7ffaec0a98f0 legacy=0x7ffaec0b5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:26.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.039+0000 7ffae27fc640 1 -- 172.21.15.44:0/1405541239 >> v1:172.21.15.44:6789/0 conn(0x7ffaec13f710 legacy=0x7ffaec13bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:26.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.039+0000 7ffae27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:26.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.039+0000 7ffae27fc640 1 monclient: found mon.c 2024-03-25T19:42:26.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.039+0000 7ffae27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:26.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.039+0000 7ffae27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:42:26.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.039+0000 7ffae27fc640 1 -- 172.21.15.44:0/1405541239 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec13a090 con 0x7ffaec1430c0 2024-03-25T19:42:26.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.039+0000 7ffae27fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:26.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.039+0000 7ffae27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:26.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.039+0000 7ffae27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:26.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.039+0000 7ffaf424e640 5 monclient: authenticate success, global_id 4526 2024-03-25T19:42:26.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.039+0000 7ffaf424e640 10 monclient: _renew_subs 2024-03-25T19:42:26.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.039+0000 7ffaf424e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:42:26.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.039+0000 7ffaf424e640 1 -- 172.21.15.44:0/1405541239 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaec139d20 con 0x7ffaec1430c0 2024-03-25T19:42:26.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.039+0000 7ffaf424e640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:26.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.039+0000 7ffaf424e640 10 monclient: _renew_subs 2024-03-25T19:42:26.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.039+0000 7ffaf424e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:42:26.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.039+0000 7ffaf424e640 1 -- 172.21.15.44:0/1405541239 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaec13a5b0 con 0x7ffaec1430c0 2024-03-25T19:42:26.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.039+0000 7ffaf424e640 1 librados: init done 2024-03-25T19:42:26.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.040+0000 7ffaf424e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:26.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.040+0000 7ffaf424e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:26.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.040+0000 7ffaf424e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:42:26.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.040+0000 7ffaf424e640 1 -- 172.21.15.44:0/1405541239 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaec14f2d0 con 0x7ffaec1430c0 2024-03-25T19:42:26.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.371+0000 7ffae27fc640 1 -- 172.21.15.44:0/1405541239 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffae4002050 con 0x7ffaec1430c0 2024-03-25T19:42:26.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.371+0000 7ffae27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:26.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.371+0000 7ffae27fc640 1 -- 172.21.15.44:0/1405541239 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ffae40015d0 con 0x7ffaec1430c0 2024-03-25T19:42:26.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.371+0000 7ffae27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:26.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.371+0000 7ffae27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:26.371 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.371+0000 7ffae27fc640 10 monclient: dump: 2024-03-25T19:42:26.371 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:26.371 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:26.371 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:26.372 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:26.372 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:26.372 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:26.372 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:26.372 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:26.372 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:26.372 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:26.372 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.371+0000 7ffae3fff640 4 set_mon_vals no callback set 2024-03-25T19:42:26.373 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.371+0000 7ffae27fc640 1 -- 172.21.15.44:0/1405541239 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7ffae401ae20 con 0x7ffaec1430c0 2024-03-25T19:42:26.373 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.372+0000 7ffaf1fea640 10 monclient: get_auth_request con 0x7ffad006e110 auth_method 0 2024-03-25T19:42:26.373 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.372+0000 7ffae27fc640 10 client.4526.objecter ms_handle_connect 0x7ffad006e110 2024-03-25T19:42:26.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.378+0000 7ffae27fc640 1 -- 172.21.15.44:0/1405541239 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7ffae4054a50 con 0x7ffaec1430c0 2024-03-25T19:42:26.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.378+0000 7ffae27fc640 10 client.4526.objecter ms_dispatch 0x7ffaec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:26.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.378+0000 7ffae27fc640 3 client.4526.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:26.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.378+0000 7ffae27fc640 3 client.4526.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:26.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.378+0000 7ffae27fc640 20 client.4526.objecter dump_active .. 0 homeless 2024-03-25T19:42:26.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.378+0000 7ffae27fc640 1 -- 172.21.15.44:0/1405541239 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffae4050560 con 0x7ffaec1430c0 2024-03-25T19:42:26.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.378+0000 7ffae27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:26.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.378+0000 7ffae27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:26.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.611+0000 7ffaf424e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:26.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.611+0000 7ffaf424e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:26.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.611+0000 7ffaf424e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:42:26.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.611+0000 7ffaf424e640 1 -- 172.21.15.44:0/1405541239 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffaec13a220 con 0x7ffaec1430c0 2024-03-25T19:42:26.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.655+0000 7ffae27fc640 1 -- 172.21.15.44:0/1405541239 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7ffae4052490 con 0x7ffaec1430c0 2024-03-25T19:42:26.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.655+0000 7ffae27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:26.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.655+0000 7ffae27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:26.656 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:42:26.656 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:26.658 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.658+0000 7ffaf424e640 10 librados: watch_flush enter 2024-03-25T19:42:26.659 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.658+0000 7ffaf424e640 10 librados: watch_flush exit 2024-03-25T19:42:26.659 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.658+0000 7ffaf424e640 20 client.4526.objecter shutdown clearing up homeless session... 2024-03-25T19:42:26.659 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.658+0000 7ffaf424e640 10 client.4526.objecter successfully canceled tick 2024-03-25T19:42:26.659 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.658+0000 7ffaf424e640 1 -- 172.21.15.44:0/1405541239 >> v1:172.21.15.111:6800/3883215793 conn(0x7ffad006e110 legacy=0x7ffad00705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:26.659 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.658+0000 7ffaf424e640 10 monclient: shutdown 2024-03-25T19:42:26.659 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.658+0000 7ffaf424e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:26.659 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.658+0000 7ffaf424e640 1 -- 172.21.15.44:0/1405541239 >> v1:172.21.15.44:6790/0 conn(0x7ffaec1430c0 legacy=0x7ffaec13c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:26.659 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.658+0000 7ffaf424e640 1 -- 172.21.15.44:0/1405541239 shutdown_connections 2024-03-25T19:42:26.659 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.658+0000 7ffaf424e640 1 -- 172.21.15.44:0/1405541239 >> 172.21.15.44:0/1405541239 conn(0x7ffaec0b7ea0 msgr2=0x7ffaec074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:26.659 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.658+0000 7ffaf424e640 1 -- 172.21.15.44:0/1405541239 shutdown_connections 2024-03-25T19:42:26.659 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.658+0000 7ffaf424e640 1 -- 172.21.15.44:0/1405541239 wait complete. 2024-03-25T19:42:26.659 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:26.658+0000 7ffaf424e640 1 librados: shutdown 2024-03-25T19:42:26.669 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:33.476 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:33.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.584+0000 7fd5a4afb640 10 monclient: get_monmap_and_config 2024-03-25T19:42:33.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.584+0000 7fd5a4afb640 10 monclient: build_initial_monmap 2024-03-25T19:42:33.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.584+0000 7fd5a4afb640 10 monclient: monmap: 2024-03-25T19:42:33.583 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:42:33.584 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:33.584 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:42:33.584686+0000 2024-03-25T19:42:33.584 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:42:33.584686+0000 2024-03-25T19:42:33.584 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:33.584 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:33.584 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:42:33.584 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:42:33.584 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:42:33.584 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:33.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.585+0000 7fd5a4afb640 1 Processor -- start 2024-03-25T19:42:33.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.585+0000 7fd5a4afb640 1 -- start start 2024-03-25T19:42:33.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.585+0000 7fd5a4afb640 10 monclient: init 2024-03-25T19:42:33.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.586+0000 7fd5a4afb640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:33.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.586+0000 7fd5a4afb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:33.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.586+0000 7fd5a4afb640 10 monclient(hunting): picked mon.noname-c con 0x7fd5a00a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:42:33.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.586+0000 7fd5a4afb640 10 monclient(hunting): picked mon.noname-b con 0x7fd5a013f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:42:33.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.586+0000 7fd5a4afb640 10 monclient(hunting): picked mon.noname-a con 0x7fd5a01430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:42:33.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.586+0000 7fd5a4afb640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a0142520 con 0x7fd5a01430c0 2024-03-25T19:42:33.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.586+0000 7fd5a4afb640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a00b28a0 con 0x7fd5a00a98f0 2024-03-25T19:42:33.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.586+0000 7fd5a4afb640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a00b2a30 con 0x7fd5a013f710 2024-03-25T19:42:33.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.586+0000 7fd5a4afb640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:33.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.586+0000 7fd5a4afb640 10 monclient(hunting): authenticate will time out at 5817.756059s 2024-03-25T19:42:33.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.593+0000 7fd597fff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fd5a013f710 0x7fd5a01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:39800/0 (socket says 172.21.15.44:39800) 2024-03-25T19:42:33.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.593+0000 7fd597fff640 1 -- 172.21.15.44:0/1304521662 learned_addr learned my addr 172.21.15.44:0/1304521662 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:42:33.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.608+0000 7fd59dd9b640 1 -- 172.21.15.44:0/1304521662 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2601392706 0 0) 0x7fd5a00b2a30 con 0x7fd5a013f710 2024-03-25T19:42:33.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.608+0000 7fd59dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:33.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.608+0000 7fd59dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:33.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.608+0000 7fd59dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:33.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.608+0000 7fd59dd9b640 10 monclient(hunting): my global_id is 4529 2024-03-25T19:42:33.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.608+0000 7fd59dd9b640 1 -- 172.21.15.44:0/1304521662 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd58c003520 con 0x7fd5a013f710 2024-03-25T19:42:33.618 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.620+0000 7fd59dd9b640 1 -- 172.21.15.44:0/1304521662 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2452994153 0 0) 0x7fd5a0142520 con 0x7fd5a01430c0 2024-03-25T19:42:33.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.620+0000 7fd59dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:33.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.620+0000 7fd59dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:33.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.620+0000 7fd59dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:33.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.620+0000 7fd59dd9b640 10 monclient(hunting): my global_id is 4554 2024-03-25T19:42:33.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.620+0000 7fd59dd9b640 1 -- 172.21.15.44:0/1304521662 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5a00b2a30 con 0x7fd5a01430c0 2024-03-25T19:42:33.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.622+0000 7fd59dd9b640 1 -- 172.21.15.44:0/1304521662 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2226170940 0 0) 0x7fd58c003520 con 0x7fd5a013f710 2024-03-25T19:42:33.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.622+0000 7fd59dd9b640 1 -- 172.21.15.44:0/1304521662 >> v1:172.21.15.111:6789/0 conn(0x7fd5a00a98f0 legacy=0x7fd5a00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:33.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.622+0000 7fd59dd9b640 1 -- 172.21.15.44:0/1304521662 >> v1:172.21.15.44:6789/0 conn(0x7fd5a01430c0 legacy=0x7fd5a014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:33.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.622+0000 7fd59dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:42:33.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.622+0000 7fd59dd9b640 1 monclient: found mon.noname-b 2024-03-25T19:42:33.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.622+0000 7fd59dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:33.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.622+0000 7fd59dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:42:33.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.622+0000 7fd59dd9b640 1 -- 172.21.15.44:0/1304521662 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a00b44c0 con 0x7fd5a013f710 2024-03-25T19:42:33.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.622+0000 7fd59dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:42:33.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.622+0000 7fd59dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:42:33.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.622+0000 7fd59dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:33.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.622+0000 7fd59dd9b640 1 -- 172.21.15.44:0/1304521662 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fd588001690 con 0x7fd5a013f710 2024-03-25T19:42:33.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.622+0000 7fd59dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:33.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.622+0000 7fd59dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:33.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.622+0000 7fd59dd9b640 10 monclient: dump: 2024-03-25T19:42:33.622 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:33.622 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:33.622 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:33.622 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:33.622 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:33.622 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:33.622 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:33.623 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:33.623 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:33.623 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:33.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.622+0000 7fd5a4afb640 5 monclient: authenticate success, global_id 4529 2024-03-25T19:42:33.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.622+0000 7fd5a4afb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:33.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.630+0000 7fd59dd9b640 1 -- 172.21.15.44:0/1304521662 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5880017f0 con 0x7fd5a013f710 2024-03-25T19:42:33.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.630+0000 7fd59dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:33.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.630+0000 7fd59dd9b640 1 -- 172.21.15.44:0/1304521662 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fd588001c00 con 0x7fd5a013f710 2024-03-25T19:42:33.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.630+0000 7fd5a4afb640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:33.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.630+0000 7fd5a4afb640 4 set_mon_vals no callback set 2024-03-25T19:42:33.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.630+0000 7fd59dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:33.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.630+0000 7fd59dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:33.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.630+0000 7fd59dd9b640 10 monclient: dump: 2024-03-25T19:42:33.630 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:33.630 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:33.630 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:33.630 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:33.630 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:33.630 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:33.630 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:33.630 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:33.630 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:33.630 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:33.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.631+0000 7fd5a4afb640 10 monclient: shutdown 2024-03-25T19:42:33.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.631+0000 7fd5a4afb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:33.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.631+0000 7fd5a4afb640 1 -- 172.21.15.44:0/1304521662 >> v1:172.21.15.44:6790/0 conn(0x7fd5a013f710 legacy=0x7fd5a01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:33.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.631+0000 7fd5a4afb640 1 -- 172.21.15.44:0/1304521662 shutdown_connections 2024-03-25T19:42:33.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.631+0000 7fd5a4afb640 1 -- 172.21.15.44:0/1304521662 >> 172.21.15.44:0/1304521662 conn(0x7fd5a00b7ea0 msgr2=0x7fd5a00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:33.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.631+0000 7fd5a4afb640 1 -- 172.21.15.44:0/1304521662 shutdown_connections 2024-03-25T19:42:33.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.631+0000 7fd5a4afb640 1 -- 172.21.15.44:0/1304521662 wait complete. 2024-03-25T19:42:33.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.632+0000 7fd5a4afb640 10 monclient: build_initial_monmap 2024-03-25T19:42:33.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.632+0000 7fd5a4afb640 10 monclient: monmap: 2024-03-25T19:42:33.632 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:42:33.632 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:33.633 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:42:33.633 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:42:33.633 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:33.633 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:42:33.633 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:42:33.633 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:42:33.633 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:42:33.633 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:33.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.632+0000 7fd5a4afb640 1 librados: starting msgr at 2024-03-25T19:42:33.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.632+0000 7fd5a4afb640 1 librados: starting objecter 2024-03-25T19:42:33.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.633+0000 7fd5a4afb640 1 Processor -- start 2024-03-25T19:42:33.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.633+0000 7fd5a4afb640 1 -- start start 2024-03-25T19:42:33.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.633+0000 7fd5a4afb640 1 librados: setting wanted keys 2024-03-25T19:42:33.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.633+0000 7fd5a4afb640 1 librados: calling monclient init 2024-03-25T19:42:33.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.633+0000 7fd5a4afb640 10 monclient: init 2024-03-25T19:42:33.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.634+0000 7fd5a4afb640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:33.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.634+0000 7fd5a4afb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:33.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.634+0000 7fd5a4afb640 10 monclient(hunting): picked mon.noname-c con 0x7fd5a00a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T19:42:33.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.634+0000 7fd5a4afb640 10 monclient(hunting): picked mon.noname-b con 0x7fd5a013f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:42:33.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.634+0000 7fd5a4afb640 10 monclient(hunting): picked mon.noname-a con 0x7fd5a01430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:42:33.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.634+0000 7fd5a4afb640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a013be70 con 0x7fd5a01430c0 2024-03-25T19:42:33.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.634+0000 7fd5a4afb640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a0131660 con 0x7fd5a013f710 2024-03-25T19:42:33.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.634+0000 7fd5a4afb640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a01317f0 con 0x7fd5a00a98f0 2024-03-25T19:42:33.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.634+0000 7fd5a4afb640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:33.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.634+0000 7fd5a4afb640 10 monclient(hunting): authenticate will time out at 5817.804203s 2024-03-25T19:42:33.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.635+0000 7fd59e59c640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fd5a00a98f0 0x7fd5a0130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:39860/0 (socket says 172.21.15.44:39860) 2024-03-25T19:42:33.635 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.635+0000 7fd59e59c640 1 -- 172.21.15.44:0/889598735 learned_addr learned my addr 172.21.15.44:0/889598735 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:42:33.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.639+0000 7fd596ffd640 10 client.?.objecter ms_handle_connect 0x7fd5a00a98f0 2024-03-25T19:42:33.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.639+0000 7fd596ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:33.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.650+0000 7fd596ffd640 10 client.?.objecter ms_handle_connect 0x7fd5a01430c0 2024-03-25T19:42:33.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.650+0000 7fd596ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:33.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.652+0000 7fd596ffd640 1 -- 172.21.15.44:0/889598735 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906047221 0 0) 0x7fd5a01317f0 con 0x7fd5a00a98f0 2024-03-25T19:42:33.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.652+0000 7fd596ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:33.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.652+0000 7fd596ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:33.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.652+0000 7fd596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:33.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.652+0000 7fd596ffd640 10 monclient(hunting): my global_id is 4532 2024-03-25T19:42:33.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.652+0000 7fd596ffd640 1 -- 172.21.15.44:0/889598735 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd580004eb0 con 0x7fd5a00a98f0 2024-03-25T19:42:33.652 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.653+0000 7fd596ffd640 10 client.?.objecter ms_handle_connect 0x7fd5a013f710 2024-03-25T19:42:33.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.653+0000 7fd596ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:33.668 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.669+0000 7fd596ffd640 1 -- 172.21.15.44:0/889598735 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 644897509 0 0) 0x7fd5a013be70 con 0x7fd5a01430c0 2024-03-25T19:42:33.668 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.669+0000 7fd596ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:33.668 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.669+0000 7fd596ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:33.668 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.669+0000 7fd596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:33.668 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.669+0000 7fd596ffd640 10 monclient(hunting): my global_id is 4557 2024-03-25T19:42:33.668 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.669+0000 7fd596ffd640 1 -- 172.21.15.44:0/889598735 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5a01317f0 con 0x7fd5a01430c0 2024-03-25T19:42:33.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.669+0000 7fd596ffd640 1 -- 172.21.15.44:0/889598735 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706973373 0 0) 0x7fd5a0131660 con 0x7fd5a013f710 2024-03-25T19:42:33.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.669+0000 7fd596ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:33.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.669+0000 7fd596ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:33.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.669+0000 7fd596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:33.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.669+0000 7fd596ffd640 10 monclient(hunting): my global_id is 4543 2024-03-25T19:42:33.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.669+0000 7fd596ffd640 1 -- 172.21.15.44:0/889598735 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5a013be70 con 0x7fd5a013f710 2024-03-25T19:42:33.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.670+0000 7fd596ffd640 1 -- 172.21.15.44:0/889598735 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2894536496 0 0) 0x7fd580004eb0 con 0x7fd5a00a98f0 2024-03-25T19:42:33.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.670+0000 7fd596ffd640 1 -- 172.21.15.44:0/889598735 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5a0131660 con 0x7fd5a00a98f0 2024-03-25T19:42:33.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.670+0000 7fd596ffd640 1 -- 172.21.15.44:0/889598735 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fd5900019d0 con 0x7fd5a00a98f0 2024-03-25T19:42:33.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.670+0000 7fd596ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:33.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.670+0000 7fd596ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:33.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.670+0000 7fd596ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:33.670 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:33.671 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:33.671 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:33.671 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:33.671 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:33.671 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:33.671 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:33.671 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:33.671 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:33.671 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:33.671 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.670+0000 7fd596ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:33.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.677+0000 7fd596ffd640 1 -- 172.21.15.44:0/889598735 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1721018946 0 0) 0x7fd5a0131660 con 0x7fd5a00a98f0 2024-03-25T19:42:33.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.677+0000 7fd596ffd640 1 -- 172.21.15.44:0/889598735 >> v1:172.21.15.111:6789/0 conn(0x7fd5a013f710 legacy=0x7fd5a01d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:33.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.677+0000 7fd596ffd640 1 -- 172.21.15.44:0/889598735 >> v1:172.21.15.44:6789/0 conn(0x7fd5a01430c0 legacy=0x7fd5a01e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:33.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.677+0000 7fd596ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:33.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.677+0000 7fd596ffd640 1 monclient: found mon.c 2024-03-25T19:42:33.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.677+0000 7fd596ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:33.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.677+0000 7fd596ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:42:33.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.677+0000 7fd596ffd640 1 -- 172.21.15.44:0/889598735 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a0131980 con 0x7fd5a00a98f0 2024-03-25T19:42:33.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.677+0000 7fd596ffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:33.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.677+0000 7fd596ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:33.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.677+0000 7fd596ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:33.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.677+0000 7fd5a4afb640 5 monclient: authenticate success, global_id 4532 2024-03-25T19:42:33.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.677+0000 7fd5a4afb640 10 monclient: _renew_subs 2024-03-25T19:42:33.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.677+0000 7fd5a4afb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:42:33.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.677+0000 7fd5a4afb640 1 -- 172.21.15.44:0/889598735 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a0135c60 con 0x7fd5a00a98f0 2024-03-25T19:42:33.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.677+0000 7fd5a4afb640 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:33.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.677+0000 7fd5a4afb640 10 monclient: _renew_subs 2024-03-25T19:42:33.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.677+0000 7fd5a4afb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:42:33.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.677+0000 7fd5a4afb640 1 -- 172.21.15.44:0/889598735 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a00b7770 con 0x7fd5a00a98f0 2024-03-25T19:42:33.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.677+0000 7fd5a4afb640 1 librados: init done 2024-03-25T19:42:33.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.679+0000 7fd5a4afb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:33.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.679+0000 7fd5a4afb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:33.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.679+0000 7fd5a4afb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:42:33.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.679+0000 7fd5a4afb640 1 -- 172.21.15.44:0/889598735 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a01371d0 con 0x7fd5a00a98f0 2024-03-25T19:42:33.682 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.683+0000 7fd596ffd640 1 -- 172.21.15.44:0/889598735 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5900035f0 con 0x7fd5a00a98f0 2024-03-25T19:42:33.682 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.683+0000 7fd596ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:33.682 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:33.683+0000 7fd59d59a640 4 set_mon_vals no callback set 2024-03-25T19:42:34.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.003+0000 7fd596ffd640 1 -- 172.21.15.44:0/889598735 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fd590003a00 con 0x7fd5a00a98f0 2024-03-25T19:42:34.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.003+0000 7fd596ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:34.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.003+0000 7fd596ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:34.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.003+0000 7fd596ffd640 10 monclient: dump: 2024-03-25T19:42:34.002 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:34.002 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:34.002 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:34.002 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:34.002 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:34.002 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:34.002 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:34.002 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:34.002 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:34.003 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:34.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.003+0000 7fd596ffd640 1 -- 172.21.15.44:0/889598735 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7fd590003be0 con 0x7fd5a00a98f0 2024-03-25T19:42:34.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.004+0000 7fd597fff640 10 monclient: get_auth_request con 0x7fd58006de30 auth_method 0 2024-03-25T19:42:34.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.004+0000 7fd596ffd640 10 client.4532.objecter ms_handle_connect 0x7fd58006de30 2024-03-25T19:42:34.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.013+0000 7fd596ffd640 1 -- 172.21.15.44:0/889598735 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7fd590054850 con 0x7fd5a00a98f0 2024-03-25T19:42:34.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.013+0000 7fd596ffd640 10 client.4532.objecter ms_dispatch 0x7fd5a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:34.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.013+0000 7fd596ffd640 3 client.4532.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:34.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.013+0000 7fd596ffd640 3 client.4532.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:34.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.013+0000 7fd596ffd640 20 client.4532.objecter dump_active .. 0 homeless 2024-03-25T19:42:34.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.013+0000 7fd596ffd640 1 -- 172.21.15.44:0/889598735 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd590050360 con 0x7fd5a00a98f0 2024-03-25T19:42:34.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.013+0000 7fd596ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:34.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.013+0000 7fd596ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:34.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.271+0000 7fd5a4afb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:34.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.271+0000 7fd5a4afb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:34.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.271+0000 7fd5a4afb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:42:34.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.271+0000 7fd5a4afb640 1 -- 172.21.15.44:0/889598735 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5a00627b0 con 0x7fd5a00a98f0 2024-03-25T19:42:34.302 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.302+0000 7fd596ffd640 1 -- 172.21.15.44:0/889598735 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7fd590052290 con 0x7fd5a00a98f0 2024-03-25T19:42:34.302 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.302+0000 7fd596ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:34.302 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.302+0000 7fd596ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:34.302 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:42:34.302 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:34.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.304+0000 7fd5a4afb640 10 librados: watch_flush enter 2024-03-25T19:42:34.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.304+0000 7fd5a4afb640 10 librados: watch_flush exit 2024-03-25T19:42:34.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.304+0000 7fd5a4afb640 20 client.4532.objecter shutdown clearing up homeless session... 2024-03-25T19:42:34.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.304+0000 7fd5a4afb640 10 client.4532.objecter successfully canceled tick 2024-03-25T19:42:34.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.305+0000 7fd5a4afb640 1 -- 172.21.15.44:0/889598735 >> v1:172.21.15.111:6800/3883215793 conn(0x7fd58006de30 legacy=0x7fd5800702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:34.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.305+0000 7fd5a4afb640 10 monclient: shutdown 2024-03-25T19:42:34.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.305+0000 7fd5a4afb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:34.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.305+0000 7fd5a4afb640 1 -- 172.21.15.44:0/889598735 >> v1:172.21.15.44:6790/0 conn(0x7fd5a00a98f0 legacy=0x7fd5a0130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:34.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.305+0000 7fd5a4afb640 1 -- 172.21.15.44:0/889598735 shutdown_connections 2024-03-25T19:42:34.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.305+0000 7fd5a4afb640 1 -- 172.21.15.44:0/889598735 >> 172.21.15.44:0/889598735 conn(0x7fd5a00b7ea0 msgr2=0x7fd5a0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:34.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.305+0000 7fd5a4afb640 1 -- 172.21.15.44:0/889598735 shutdown_connections 2024-03-25T19:42:34.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.305+0000 7fd5a4afb640 1 -- 172.21.15.44:0/889598735 wait complete. 2024-03-25T19:42:34.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:34.305+0000 7fd5a4afb640 1 librados: shutdown 2024-03-25T19:42:34.316 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:39.927 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:07:30.829 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:39.979 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:07:30.881 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:40.004 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:07:30.905 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:40.041 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:07:30.921 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:40.042 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:07:30.943 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:40.049 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:07:30.951 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:40.052 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:07:30.932 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:40.084 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:07:30.986 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:40.092 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:07:30.972 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:41.122 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:41.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.230+0000 7f6a7d806640 10 monclient: get_monmap_and_config 2024-03-25T19:42:41.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.230+0000 7f6a7d806640 10 monclient: build_initial_monmap 2024-03-25T19:42:41.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.230+0000 7f6a7d806640 10 monclient: monmap: 2024-03-25T19:42:41.231 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:42:41.231 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:41.231 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:42:41.231885+0000 2024-03-25T19:42:41.231 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:42:41.231885+0000 2024-03-25T19:42:41.231 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:41.231 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:41.231 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:42:41.231 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:42:41.231 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:42:41.231 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:41.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.231+0000 7f6a7d806640 1 Processor -- start 2024-03-25T19:42:41.233 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.231+0000 7f6a7d806640 1 -- start start 2024-03-25T19:42:41.233 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.231+0000 7f6a7d806640 10 monclient: init 2024-03-25T19:42:41.233 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.232+0000 7f6a7d806640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:41.233 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.232+0000 7f6a7d806640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:41.233 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.232+0000 7f6a7d806640 10 monclient(hunting): picked mon.noname-b con 0x7f6a780705b0 addr v1:172.21.15.44:6790/0 2024-03-25T19:42:41.233 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.232+0000 7f6a7d806640 10 monclient(hunting): picked mon.noname-c con 0x7f6a781402d0 addr v1:172.21.15.111:6789/0 2024-03-25T19:42:41.233 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.232+0000 7f6a7d806640 10 monclient(hunting): picked mon.noname-a con 0x7f6a78149ae0 addr v1:172.21.15.44:6789/0 2024-03-25T19:42:41.233 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.232+0000 7f6a7d806640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a780718c0 con 0x7f6a78149ae0 2024-03-25T19:42:41.233 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.232+0000 7f6a7d806640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a780714c0 con 0x7f6a781402d0 2024-03-25T19:42:41.233 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.232+0000 7f6a7d806640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a78072b50 con 0x7f6a780705b0 2024-03-25T19:42:41.233 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.232+0000 7f6a7d806640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:41.233 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.232+0000 7f6a7d806640 10 monclient(hunting): authenticate will time out at 5825.403270s 2024-03-25T19:42:41.236 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.236+0000 7f6a76ffd640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f6a780705b0 0x7f6a7813d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:44800/0 (socket says 172.21.15.44:44800) 2024-03-25T19:42:41.236 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.236+0000 7f6a76ffd640 1 -- 172.21.15.44:0/1214147962 learned_addr learned my addr 172.21.15.44:0/1214147962 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:42:41.253 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.253+0000 7f6a75ffb640 1 -- 172.21.15.44:0/1214147962 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2175041499 0 0) 0x7f6a78072b50 con 0x7f6a780705b0 2024-03-25T19:42:41.253 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.253+0000 7f6a75ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:41.253 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.253+0000 7f6a75ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:41.254 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.253+0000 7f6a75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:41.254 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.253+0000 7f6a75ffb640 10 monclient(hunting): my global_id is 4535 2024-03-25T19:42:41.254 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.253+0000 7f6a75ffb640 1 -- 172.21.15.44:0/1214147962 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a60003520 con 0x7f6a780705b0 2024-03-25T19:42:41.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.259+0000 7f6a75ffb640 1 -- 172.21.15.44:0/1214147962 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 24408417 0 0) 0x7f6a780718c0 con 0x7f6a78149ae0 2024-03-25T19:42:41.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.259+0000 7f6a75ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:41.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.259+0000 7f6a75ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:41.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.259+0000 7f6a75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:41.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.259+0000 7f6a75ffb640 10 monclient(hunting): my global_id is 4560 2024-03-25T19:42:41.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.259+0000 7f6a75ffb640 1 -- 172.21.15.44:0/1214147962 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a78072b50 con 0x7f6a78149ae0 2024-03-25T19:42:41.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.259+0000 7f6a75ffb640 1 -- 172.21.15.44:0/1214147962 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2138949929 0 0) 0x7f6a780714c0 con 0x7f6a781402d0 2024-03-25T19:42:41.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.259+0000 7f6a75ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:41.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.259+0000 7f6a75ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:41.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.259+0000 7f6a75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:41.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.259+0000 7f6a75ffb640 10 monclient(hunting): my global_id is 4546 2024-03-25T19:42:41.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.259+0000 7f6a75ffb640 1 -- 172.21.15.44:0/1214147962 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a780718c0 con 0x7f6a781402d0 2024-03-25T19:42:41.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.265+0000 7f6a75ffb640 1 -- 172.21.15.44:0/1214147962 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 8330071 0 0) 0x7f6a78072b50 con 0x7f6a78149ae0 2024-03-25T19:42:41.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.265+0000 7f6a75ffb640 1 -- 172.21.15.44:0/1214147962 >> v1:172.21.15.44:6790/0 conn(0x7f6a780705b0 legacy=0x7f6a7813d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:41.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.265+0000 7f6a75ffb640 1 -- 172.21.15.44:0/1214147962 >> v1:172.21.15.111:6789/0 conn(0x7f6a781402d0 legacy=0x7f6a78140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:41.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.265+0000 7f6a75ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:42:41.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.265+0000 7f6a75ffb640 1 monclient: found mon.noname-a 2024-03-25T19:42:41.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.265+0000 7f6a75ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:41.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.265+0000 7f6a75ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:42:41.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.265+0000 7f6a75ffb640 1 -- 172.21.15.44:0/1214147962 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7813dd10 con 0x7f6a78149ae0 2024-03-25T19:42:41.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.265+0000 7f6a75ffb640 10 monclient: _finish_auth 0 2024-03-25T19:42:41.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.265+0000 7f6a75ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:42:41.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.265+0000 7f6a75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:41.267 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.265+0000 7f6a75ffb640 1 -- 172.21.15.44:0/1214147962 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6a700019f0 con 0x7f6a78149ae0 2024-03-25T19:42:41.267 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.265+0000 7f6a75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:41.267 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.265+0000 7f6a75ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:41.267 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.265+0000 7f6a75ffb640 10 monclient: dump: 2024-03-25T19:42:41.267 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:41.267 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:41.267 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:41.267 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:41.267 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:41.267 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:41.267 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:41.267 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:41.267 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:41.267 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:41.267 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.265+0000 7f6a7d806640 5 monclient: authenticate success, global_id 4560 2024-03-25T19:42:41.268 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.265+0000 7f6a7d806640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:41.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.275+0000 7f6a75ffb640 1 -- 172.21.15.44:0/1214147962 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a70001b50 con 0x7f6a78149ae0 2024-03-25T19:42:41.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.275+0000 7f6a75ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:41.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.275+0000 7f6a75ffb640 1 -- 172.21.15.44:0/1214147962 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6a70001f60 con 0x7f6a78149ae0 2024-03-25T19:42:41.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.275+0000 7f6a75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:41.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.275+0000 7f6a75ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:41.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.275+0000 7f6a75ffb640 10 monclient: dump: 2024-03-25T19:42:41.276 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:41.276 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:41.276 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:41.276 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:41.276 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:41.276 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:41.276 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:41.276 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:41.277 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:41.277 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:41.277 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.275+0000 7f6a7d806640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:41.277 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.275+0000 7f6a7d806640 4 set_mon_vals no callback set 2024-03-25T19:42:41.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.276+0000 7f6a7d806640 10 monclient: shutdown 2024-03-25T19:42:41.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.276+0000 7f6a7d806640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:41.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.276+0000 7f6a7d806640 1 -- 172.21.15.44:0/1214147962 >> v1:172.21.15.44:6789/0 conn(0x7f6a78149ae0 legacy=0x7f6a7814bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:41.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.276+0000 7f6a7d806640 1 -- 172.21.15.44:0/1214147962 shutdown_connections 2024-03-25T19:42:41.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.276+0000 7f6a7d806640 1 -- 172.21.15.44:0/1214147962 >> 172.21.15.44:0/1214147962 conn(0x7f6a780b5d10 msgr2=0x7f6a780b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:41.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.276+0000 7f6a7d806640 1 -- 172.21.15.44:0/1214147962 shutdown_connections 2024-03-25T19:42:41.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.276+0000 7f6a7d806640 1 -- 172.21.15.44:0/1214147962 wait complete. 2024-03-25T19:42:41.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.277+0000 7f6a7d806640 10 monclient: build_initial_monmap 2024-03-25T19:42:41.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.277+0000 7f6a7d806640 10 monclient: monmap: 2024-03-25T19:42:41.279 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:42:41.279 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:41.279 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:42:41.279 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:42:41.279 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:41.279 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:42:41.279 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:42:41.279 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:42:41.279 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:42:41.279 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:41.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.278+0000 7f6a7d806640 1 librados: starting msgr at 2024-03-25T19:42:41.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.278+0000 7f6a7d806640 1 librados: starting objecter 2024-03-25T19:42:41.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.278+0000 7f6a7d806640 1 Processor -- start 2024-03-25T19:42:41.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.278+0000 7f6a7d806640 1 -- start start 2024-03-25T19:42:41.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.278+0000 7f6a7d806640 1 librados: setting wanted keys 2024-03-25T19:42:41.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.278+0000 7f6a7d806640 1 librados: calling monclient init 2024-03-25T19:42:41.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.278+0000 7f6a7d806640 10 monclient: init 2024-03-25T19:42:41.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.279+0000 7f6a7d806640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:41.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.279+0000 7f6a7d806640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:41.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.279+0000 7f6a7d806640 10 monclient(hunting): picked mon.noname-a con 0x7f6a780705b0 addr v1:172.21.15.44:6789/0 2024-03-25T19:42:41.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.279+0000 7f6a7d806640 10 monclient(hunting): picked mon.noname-c con 0x7f6a781402d0 addr v1:172.21.15.44:6790/0 2024-03-25T19:42:41.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.279+0000 7f6a7d806640 10 monclient(hunting): picked mon.noname-b con 0x7f6a78149ae0 addr v1:172.21.15.111:6789/0 2024-03-25T19:42:41.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.279+0000 7f6a7d806640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a7806dc50 con 0x7f6a780705b0 2024-03-25T19:42:41.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.279+0000 7f6a7d806640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a7806d7c0 con 0x7f6a78149ae0 2024-03-25T19:42:41.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.279+0000 7f6a7d806640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a78134870 con 0x7f6a781402d0 2024-03-25T19:42:41.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.279+0000 7f6a7d806640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:41.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.279+0000 7f6a7d806640 10 monclient(hunting): authenticate will time out at 5825.450300s 2024-03-25T19:42:41.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.290+0000 7f6a76ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f6a780705b0 0x7f6a781da800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:42898/0 (socket says 172.21.15.44:42898) 2024-03-25T19:42:41.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.290+0000 7f6a76ffd640 1 -- 172.21.15.44:0/3142627020 learned_addr learned my addr 172.21.15.44:0/3142627020 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:42:41.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.293+0000 7f6a5ffff640 10 client.?.objecter ms_handle_connect 0x7f6a780705b0 2024-03-25T19:42:41.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.293+0000 7f6a5ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:41.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.296+0000 7f6a5ffff640 10 client.?.objecter ms_handle_connect 0x7f6a781402d0 2024-03-25T19:42:41.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.296+0000 7f6a5ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:41.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.301+0000 7f6a5ffff640 10 client.?.objecter ms_handle_connect 0x7f6a78149ae0 2024-03-25T19:42:41.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.301+0000 7f6a5ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:41.302 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.302+0000 7f6a5ffff640 1 -- 172.21.15.44:0/3142627020 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977038485 0 0) 0x7f6a78134870 con 0x7f6a781402d0 2024-03-25T19:42:41.302 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.302+0000 7f6a5ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:41.302 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.302+0000 7f6a5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:41.302 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.302+0000 7f6a5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:41.302 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.302+0000 7f6a5ffff640 10 monclient(hunting): my global_id is 4538 2024-03-25T19:42:41.302 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.302+0000 7f6a5ffff640 1 -- 172.21.15.44:0/3142627020 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a64004eb0 con 0x7f6a781402d0 2024-03-25T19:42:41.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.303+0000 7f6a5ffff640 1 -- 172.21.15.44:0/3142627020 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654838841 0 0) 0x7f6a60003520 con 0x7f6a780705b0 2024-03-25T19:42:41.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.303+0000 7f6a5ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:41.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.303+0000 7f6a5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:41.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.303+0000 7f6a5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:41.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.303+0000 7f6a5ffff640 10 monclient(hunting): my global_id is 4563 2024-03-25T19:42:41.303 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.303+0000 7f6a5ffff640 1 -- 172.21.15.44:0/3142627020 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a78134870 con 0x7f6a780705b0 2024-03-25T19:42:41.319 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.319+0000 7f6a5ffff640 1 -- 172.21.15.44:0/3142627020 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3629233853 0 0) 0x7f6a7806d7c0 con 0x7f6a78149ae0 2024-03-25T19:42:41.319 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.319+0000 7f6a5ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:41.319 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.319+0000 7f6a5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:41.319 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.319+0000 7f6a5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:41.319 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.319+0000 7f6a5ffff640 10 monclient(hunting): my global_id is 4549 2024-03-25T19:42:41.319 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.319+0000 7f6a5ffff640 1 -- 172.21.15.44:0/3142627020 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a60003520 con 0x7f6a78149ae0 2024-03-25T19:42:41.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.321+0000 7f6a5ffff640 1 -- 172.21.15.44:0/3142627020 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 603141180 0 0) 0x7f6a64004eb0 con 0x7f6a781402d0 2024-03-25T19:42:41.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.321+0000 7f6a5ffff640 1 -- 172.21.15.44:0/3142627020 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a7806d7c0 con 0x7f6a781402d0 2024-03-25T19:42:41.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.321+0000 7f6a5ffff640 1 -- 172.21.15.44:0/3142627020 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6a6c0018f0 con 0x7f6a781402d0 2024-03-25T19:42:41.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.321+0000 7f6a5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:41.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.321+0000 7f6a5ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:41.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.321+0000 7f6a5ffff640 10 monclient(hunting): dump: 2024-03-25T19:42:41.322 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:41.322 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:41.322 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:41.322 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:41.322 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:41.323 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:41.323 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:41.323 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:41.323 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:41.323 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:41.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.322+0000 7f6a5ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:41.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.329+0000 7f6a5ffff640 1 -- 172.21.15.44:0/3142627020 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1366386852 0 0) 0x7f6a78134870 con 0x7f6a780705b0 2024-03-25T19:42:41.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.329+0000 7f6a5ffff640 1 -- 172.21.15.44:0/3142627020 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a64004eb0 con 0x7f6a780705b0 2024-03-25T19:42:41.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.329+0000 7f6a5ffff640 1 -- 172.21.15.44:0/3142627020 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6a68001730 con 0x7f6a780705b0 2024-03-25T19:42:41.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.329+0000 7f6a5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:41.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.329+0000 7f6a5ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:41.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.329+0000 7f6a5ffff640 10 monclient(hunting): dump: 2024-03-25T19:42:41.330 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:41.330 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:41.330 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:41.330 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:41.330 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:41.330 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:41.330 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:41.330 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:41.330 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:41.330 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:41.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.338+0000 7f6a5ffff640 1 -- 172.21.15.44:0/3142627020 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 932892500 0 0) 0x7f6a7806d7c0 con 0x7f6a781402d0 2024-03-25T19:42:41.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.338+0000 7f6a5ffff640 1 -- 172.21.15.44:0/3142627020 >> v1:172.21.15.111:6789/0 conn(0x7f6a78149ae0 legacy=0x7f6a78134160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:41.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.338+0000 7f6a5ffff640 1 -- 172.21.15.44:0/3142627020 >> v1:172.21.15.44:6789/0 conn(0x7f6a780705b0 legacy=0x7f6a781da800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:41.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.338+0000 7f6a5ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:41.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.338+0000 7f6a5ffff640 1 monclient: found mon.c 2024-03-25T19:42:41.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.338+0000 7f6a5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:41.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.338+0000 7f6a5ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:42:41.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.338+0000 7f6a5ffff640 1 -- 172.21.15.44:0/3142627020 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a78137860 con 0x7f6a781402d0 2024-03-25T19:42:41.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.338+0000 7f6a5ffff640 10 monclient: _finish_auth 0 2024-03-25T19:42:41.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.338+0000 7f6a5ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:41.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.338+0000 7f6a5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:41.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.338+0000 7f6a7d806640 5 monclient: authenticate success, global_id 4538 2024-03-25T19:42:41.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.338+0000 7f6a7d806640 10 monclient: _renew_subs 2024-03-25T19:42:41.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.338+0000 7f6a7d806640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:42:41.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.338+0000 7f6a7d806640 1 -- 172.21.15.44:0/3142627020 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a78136660 con 0x7f6a781402d0 2024-03-25T19:42:41.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.338+0000 7f6a7d806640 10 client.4538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:41.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.338+0000 7f6a7d806640 10 monclient: _renew_subs 2024-03-25T19:42:41.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.338+0000 7f6a7d806640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:42:41.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.338+0000 7f6a7d806640 1 -- 172.21.15.44:0/3142627020 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a78135ba0 con 0x7f6a781402d0 2024-03-25T19:42:41.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.338+0000 7f6a7d806640 1 librados: init done 2024-03-25T19:42:41.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.339+0000 7f6a7d806640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:41.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.339+0000 7f6a7d806640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:41.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.339+0000 7f6a7d806640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:42:41.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.339+0000 7f6a7d806640 1 -- 172.21.15.44:0/3142627020 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a78135ea0 con 0x7f6a781402d0 2024-03-25T19:42:41.675 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.674+0000 7f6a5ffff640 1 -- 172.21.15.44:0/3142627020 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a6c003610 con 0x7f6a781402d0 2024-03-25T19:42:41.675 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.675+0000 7f6a5ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:41.675 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.675+0000 7f6a5ffff640 1 -- 172.21.15.44:0/3142627020 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6a6c003920 con 0x7f6a781402d0 2024-03-25T19:42:41.675 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.675+0000 7f6a5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:41.675 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.675+0000 7f6a5ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:41.675 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.675+0000 7f6a5ffff640 10 monclient: dump: 2024-03-25T19:42:41.675 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:41.675 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:41.676 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:41.676 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:41.676 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:41.676 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:41.676 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:41.676 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:41.676 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:41.676 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:41.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.675+0000 7f6a757fa640 4 set_mon_vals no callback set 2024-03-25T19:42:41.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.675+0000 7f6a5ffff640 1 -- 172.21.15.44:0/3142627020 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f6a6c019930 con 0x7f6a781402d0 2024-03-25T19:42:41.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.676+0000 7f6a76ffd640 10 monclient: get_auth_request con 0x7f6a6406df70 auth_method 0 2024-03-25T19:42:41.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.676+0000 7f6a5ffff640 10 client.4538.objecter ms_handle_connect 0x7f6a6406df70 2024-03-25T19:42:41.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.681+0000 7f6a5ffff640 1 -- 172.21.15.44:0/3142627020 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f6a6c053650 con 0x7f6a781402d0 2024-03-25T19:42:41.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.681+0000 7f6a5ffff640 10 client.4538.objecter ms_dispatch 0x7f6a78000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:41.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.681+0000 7f6a5ffff640 3 client.4538.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:41.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.681+0000 7f6a5ffff640 3 client.4538.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:41.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.681+0000 7f6a5ffff640 20 client.4538.objecter dump_active .. 0 homeless 2024-03-25T19:42:41.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.681+0000 7f6a5ffff640 1 -- 172.21.15.44:0/3142627020 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6a6c04ef90 con 0x7f6a781402d0 2024-03-25T19:42:41.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.681+0000 7f6a5ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:41.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.681+0000 7f6a5ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:41.942 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.942+0000 7f6a7d806640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:41.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.942+0000 7f6a7d806640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:41.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.942+0000 7f6a7d806640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:42:41.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.942+0000 7f6a7d806640 1 -- 172.21.15.44:0/3142627020 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a780627b0 con 0x7f6a781402d0 2024-03-25T19:42:41.993 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.993+0000 7f6a5ffff640 1 -- 172.21.15.44:0/3142627020 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f6a6c051090 con 0x7f6a781402d0 2024-03-25T19:42:41.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.993+0000 7f6a5ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:41.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.993+0000 7f6a5ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:41.994 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:42:41.994 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:41.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.996+0000 7f6a7d806640 10 librados: watch_flush enter 2024-03-25T19:42:41.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.996+0000 7f6a7d806640 10 librados: watch_flush exit 2024-03-25T19:42:41.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.996+0000 7f6a7d806640 20 client.4538.objecter shutdown clearing up homeless session... 2024-03-25T19:42:41.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.996+0000 7f6a7d806640 10 client.4538.objecter successfully canceled tick 2024-03-25T19:42:41.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.996+0000 7f6a7d806640 1 -- 172.21.15.44:0/3142627020 >> v1:172.21.15.111:6800/3883215793 conn(0x7f6a6406df70 legacy=0x7f6a64070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:41.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.996+0000 7f6a7d806640 10 monclient: shutdown 2024-03-25T19:42:41.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.996+0000 7f6a7d806640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:41.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.996+0000 7f6a7d806640 1 -- 172.21.15.44:0/3142627020 >> v1:172.21.15.44:6790/0 conn(0x7f6a781402d0 legacy=0x7f6a78133a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:41.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.996+0000 7f6a7d806640 1 -- 172.21.15.44:0/3142627020 shutdown_connections 2024-03-25T19:42:41.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.996+0000 7f6a7d806640 1 -- 172.21.15.44:0/3142627020 >> 172.21.15.44:0/3142627020 conn(0x7f6a780b5d10 msgr2=0x7f6a7813c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:41.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.996+0000 7f6a7d806640 1 -- 172.21.15.44:0/3142627020 shutdown_connections 2024-03-25T19:42:41.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.996+0000 7f6a7d806640 1 -- 172.21.15.44:0/3142627020 wait complete. 2024-03-25T19:42:41.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:41.996+0000 7f6a7d806640 1 librados: shutdown 2024-03-25T19:42:42.009 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:48.815 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:48.930 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.929+0000 7fd512d89640 10 monclient: get_monmap_and_config 2024-03-25T19:42:48.930 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.929+0000 7fd512d89640 10 monclient: build_initial_monmap 2024-03-25T19:42:48.930 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.929+0000 7fd512d89640 10 monclient: monmap: 2024-03-25T19:42:48.930 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:42:48.930 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:48.930 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:42:48.930980+0000 2024-03-25T19:42:48.930 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:42:48.930980+0000 2024-03-25T19:42:48.930 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:48.930 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:48.930 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:42:48.930 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:42:48.931 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:42:48.931 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:48.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.930+0000 7fd512d89640 1 Processor -- start 2024-03-25T19:42:48.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.930+0000 7fd512d89640 1 -- start start 2024-03-25T19:42:48.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.930+0000 7fd512d89640 10 monclient: init 2024-03-25T19:42:48.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.931+0000 7fd512d89640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:48.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.931+0000 7fd512d89640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:48.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.931+0000 7fd512d89640 10 monclient(hunting): picked mon.noname-c con 0x7fd50c0a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:42:48.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.931+0000 7fd512d89640 10 monclient(hunting): picked mon.noname-b con 0x7fd50c13f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:42:48.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.931+0000 7fd512d89640 10 monclient(hunting): picked mon.noname-a con 0x7fd50c1430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:42:48.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.931+0000 7fd512d89640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50c142520 con 0x7fd50c1430c0 2024-03-25T19:42:48.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.931+0000 7fd512d89640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50c0b28a0 con 0x7fd50c0a98f0 2024-03-25T19:42:48.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.931+0000 7fd512d89640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50c0b2a30 con 0x7fd50c13f710 2024-03-25T19:42:48.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.931+0000 7fd512d89640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:48.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.932+0000 7fd512d89640 10 monclient(hunting): authenticate will time out at 5833.102895s 2024-03-25T19:42:48.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.934+0000 7fd50bfff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fd50c13f710 0x7fd50c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:35124/0 (socket says 172.21.15.44:35124) 2024-03-25T19:42:48.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.934+0000 7fd50bfff640 1 -- 172.21.15.44:0/734273320 learned_addr learned my addr 172.21.15.44:0/734273320 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:42:48.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.945+0000 7fd50b7fe640 1 -- 172.21.15.44:0/734273320 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729800234 0 0) 0x7fd50c0b2a30 con 0x7fd50c13f710 2024-03-25T19:42:48.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.945+0000 7fd50b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:48.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.945+0000 7fd50b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:48.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.945+0000 7fd50b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:48.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.945+0000 7fd50b7fe640 10 monclient(hunting): my global_id is 4541 2024-03-25T19:42:48.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.945+0000 7fd50b7fe640 1 -- 172.21.15.44:0/734273320 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4fc003520 con 0x7fd50c13f710 2024-03-25T19:42:48.947 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.947+0000 7fd50b7fe640 1 -- 172.21.15.44:0/734273320 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160444256 0 0) 0x7fd50c142520 con 0x7fd50c1430c0 2024-03-25T19:42:48.947 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.947+0000 7fd50b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:48.947 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.947+0000 7fd50b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:48.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.947+0000 7fd50b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:48.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.947+0000 7fd50b7fe640 10 monclient(hunting): my global_id is 4566 2024-03-25T19:42:48.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.947+0000 7fd50b7fe640 1 -- 172.21.15.44:0/734273320 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd50c0b2a30 con 0x7fd50c1430c0 2024-03-25T19:42:48.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.953+0000 7fd50b7fe640 1 -- 172.21.15.44:0/734273320 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1143818872 0 0) 0x7fd4fc003520 con 0x7fd50c13f710 2024-03-25T19:42:48.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.953+0000 7fd50b7fe640 1 -- 172.21.15.44:0/734273320 >> v1:172.21.15.111:6789/0 conn(0x7fd50c0a98f0 legacy=0x7fd50c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:48.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.953+0000 7fd50b7fe640 1 -- 172.21.15.44:0/734273320 >> v1:172.21.15.44:6789/0 conn(0x7fd50c1430c0 legacy=0x7fd50c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:48.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.953+0000 7fd50b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:42:48.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.953+0000 7fd50b7fe640 1 monclient: found mon.noname-b 2024-03-25T19:42:48.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.953+0000 7fd50b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:48.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.953+0000 7fd50b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:42:48.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.953+0000 7fd50b7fe640 1 -- 172.21.15.44:0/734273320 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50c0b44c0 con 0x7fd50c13f710 2024-03-25T19:42:48.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.953+0000 7fd50b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:42:48.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.953+0000 7fd50b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:42:48.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.953+0000 7fd50b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:48.955 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.953+0000 7fd50b7fe640 1 -- 172.21.15.44:0/734273320 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fd4f8001690 con 0x7fd50c13f710 2024-03-25T19:42:48.955 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.953+0000 7fd50b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:48.955 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.953+0000 7fd50b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:48.955 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.953+0000 7fd50b7fe640 10 monclient: dump: 2024-03-25T19:42:48.955 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:48.955 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:48.955 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:48.955 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:48.955 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:48.955 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:48.955 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:48.955 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:48.955 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:48.955 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:48.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.953+0000 7fd512d89640 5 monclient: authenticate success, global_id 4541 2024-03-25T19:42:48.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.953+0000 7fd512d89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:48.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.972+0000 7fd50b7fe640 1 -- 172.21.15.44:0/734273320 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4f80017f0 con 0x7fd50c13f710 2024-03-25T19:42:48.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.972+0000 7fd50b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:48.973 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.972+0000 7fd50b7fe640 1 -- 172.21.15.44:0/734273320 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fd4f8001c00 con 0x7fd50c13f710 2024-03-25T19:42:48.973 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.972+0000 7fd512d89640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:48.973 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.972+0000 7fd512d89640 4 set_mon_vals no callback set 2024-03-25T19:42:48.973 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.972+0000 7fd50b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:48.973 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.972+0000 7fd50b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:48.973 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.972+0000 7fd50b7fe640 10 monclient: dump: 2024-03-25T19:42:48.973 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:48.973 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:48.973 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:48.973 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:48.973 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:48.973 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:48.973 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:48.973 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:48.974 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:48.974 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:48.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.973+0000 7fd512d89640 10 monclient: shutdown 2024-03-25T19:42:48.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.973+0000 7fd512d89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:48.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.973+0000 7fd512d89640 1 -- 172.21.15.44:0/734273320 >> v1:172.21.15.44:6790/0 conn(0x7fd50c13f710 legacy=0x7fd50c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:48.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.973+0000 7fd512d89640 1 -- 172.21.15.44:0/734273320 shutdown_connections 2024-03-25T19:42:48.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.973+0000 7fd512d89640 1 -- 172.21.15.44:0/734273320 >> 172.21.15.44:0/734273320 conn(0x7fd50c0b7ea0 msgr2=0x7fd50c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:48.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.973+0000 7fd512d89640 1 -- 172.21.15.44:0/734273320 shutdown_connections 2024-03-25T19:42:48.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.973+0000 7fd512d89640 1 -- 172.21.15.44:0/734273320 wait complete. 2024-03-25T19:42:48.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.974+0000 7fd512d89640 10 monclient: build_initial_monmap 2024-03-25T19:42:48.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.974+0000 7fd512d89640 10 monclient: monmap: 2024-03-25T19:42:48.976 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:42:48.977 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:48.977 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:42:48.977 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:42:48.977 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:48.977 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:42:48.977 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:42:48.977 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:42:48.977 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:42:48.977 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:48.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.974+0000 7fd512d89640 1 librados: starting msgr at 2024-03-25T19:42:48.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.974+0000 7fd512d89640 1 librados: starting objecter 2024-03-25T19:42:48.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.974+0000 7fd512d89640 1 Processor -- start 2024-03-25T19:42:48.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.974+0000 7fd512d89640 1 -- start start 2024-03-25T19:42:48.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.974+0000 7fd512d89640 1 librados: setting wanted keys 2024-03-25T19:42:48.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.974+0000 7fd512d89640 1 librados: calling monclient init 2024-03-25T19:42:48.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.974+0000 7fd512d89640 10 monclient: init 2024-03-25T19:42:48.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.975+0000 7fd512d89640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:48.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.975+0000 7fd512d89640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:48.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.975+0000 7fd512d89640 10 monclient(hunting): picked mon.noname-c con 0x7fd50c0a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T19:42:48.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.975+0000 7fd512d89640 10 monclient(hunting): picked mon.noname-a con 0x7fd50c13f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:42:48.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.975+0000 7fd512d89640 10 monclient(hunting): picked mon.noname-b con 0x7fd50c1430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:42:48.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.975+0000 7fd512d89640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50c13be70 con 0x7fd50c13f710 2024-03-25T19:42:48.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.975+0000 7fd512d89640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50c131660 con 0x7fd50c1430c0 2024-03-25T19:42:48.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.975+0000 7fd512d89640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50c1317f0 con 0x7fd50c0a98f0 2024-03-25T19:42:48.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.975+0000 7fd512d89640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:48.978 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.975+0000 7fd512d89640 10 monclient(hunting): authenticate will time out at 5833.146562s 2024-03-25T19:42:48.987 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.988+0000 7fd510b25640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fd50c0a98f0 0x7fd50c130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:35170/0 (socket says 172.21.15.44:35170) 2024-03-25T19:42:48.988 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.988+0000 7fd510b25640 1 -- 172.21.15.44:0/2591205847 learned_addr learned my addr 172.21.15.44:0/2591205847 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:42:48.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.991+0000 7fd5097fa640 10 client.?.objecter ms_handle_connect 0x7fd50c0a98f0 2024-03-25T19:42:48.991 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.991+0000 7fd5097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:48.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.994+0000 7fd5097fa640 10 client.?.objecter ms_handle_connect 0x7fd50c1430c0 2024-03-25T19:42:48.994 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:48.994+0000 7fd5097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:49.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.002+0000 7fd5097fa640 1 -- 172.21.15.44:0/2591205847 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 787268373 0 0) 0x7fd50c1317f0 con 0x7fd50c0a98f0 2024-03-25T19:42:49.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.002+0000 7fd5097fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:49.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.002+0000 7fd5097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:49.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.002+0000 7fd5097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:49.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.002+0000 7fd5097fa640 10 monclient(hunting): my global_id is 4544 2024-03-25T19:42:49.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.002+0000 7fd5097fa640 1 -- 172.21.15.44:0/2591205847 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4f0004eb0 con 0x7fd50c0a98f0 2024-03-25T19:42:49.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.005+0000 7fd5097fa640 10 client.?.objecter ms_handle_connect 0x7fd50c13f710 2024-03-25T19:42:49.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.005+0000 7fd5097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:49.008 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.008+0000 7fd5097fa640 1 -- 172.21.15.44:0/2591205847 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917739471 0 0) 0x7fd50c131660 con 0x7fd50c1430c0 2024-03-25T19:42:49.008 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.008+0000 7fd5097fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:49.008 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.008+0000 7fd5097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:49.008 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.008+0000 7fd5097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:49.008 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.008+0000 7fd5097fa640 10 monclient(hunting): my global_id is 4555 2024-03-25T19:42:49.008 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.008+0000 7fd5097fa640 1 -- 172.21.15.44:0/2591205847 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd50c1317f0 con 0x7fd50c1430c0 2024-03-25T19:42:49.017 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.017+0000 7fd5097fa640 1 -- 172.21.15.44:0/2591205847 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 177122037 0 0) 0x7fd4f0004eb0 con 0x7fd50c0a98f0 2024-03-25T19:42:49.018 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.017+0000 7fd5097fa640 1 -- 172.21.15.44:0/2591205847 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd50c131660 con 0x7fd50c0a98f0 2024-03-25T19:42:49.018 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.017+0000 7fd5097fa640 1 -- 172.21.15.44:0/2591205847 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fd500001a30 con 0x7fd50c0a98f0 2024-03-25T19:42:49.018 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.017+0000 7fd5097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:49.018 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.017+0000 7fd5097fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:49.018 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.017+0000 7fd5097fa640 10 monclient(hunting): dump: 2024-03-25T19:42:49.018 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:49.018 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:49.018 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:49.018 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:49.018 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:49.018 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:49.018 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:49.019 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:49.019 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:49.019 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:49.019 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.017+0000 7fd5097fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:49.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.025+0000 7fd5097fa640 1 -- 172.21.15.44:0/2591205847 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1574989878 0 0) 0x7fd50c13be70 con 0x7fd50c13f710 2024-03-25T19:42:49.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.025+0000 7fd5097fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:49.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.025+0000 7fd5097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:49.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.025+0000 7fd5097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:49.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.025+0000 7fd5097fa640 10 monclient(hunting): my global_id is 4569 2024-03-25T19:42:49.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.025+0000 7fd5097fa640 1 -- 172.21.15.44:0/2591205847 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4f0004eb0 con 0x7fd50c13f710 2024-03-25T19:42:49.037 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.037+0000 7fd5097fa640 1 -- 172.21.15.44:0/2591205847 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2103997911 0 0) 0x7fd50c1317f0 con 0x7fd50c1430c0 2024-03-25T19:42:49.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.037+0000 7fd5097fa640 1 -- 172.21.15.44:0/2591205847 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd50c13be70 con 0x7fd50c1430c0 2024-03-25T19:42:49.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.037+0000 7fd5097fa640 1 -- 172.21.15.44:0/2591205847 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fd504001b50 con 0x7fd50c1430c0 2024-03-25T19:42:49.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.037+0000 7fd5097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:49.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.037+0000 7fd5097fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:49.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.037+0000 7fd5097fa640 10 monclient(hunting): dump: 2024-03-25T19:42:49.038 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:49.038 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:49.038 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:49.038 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:49.038 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:49.038 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:49.038 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:49.038 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:49.038 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:49.039 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:49.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.050+0000 7fd5097fa640 1 -- 172.21.15.44:0/2591205847 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 461560654 0 0) 0x7fd50c131660 con 0x7fd50c0a98f0 2024-03-25T19:42:49.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.050+0000 7fd5097fa640 1 -- 172.21.15.44:0/2591205847 >> v1:172.21.15.111:6789/0 conn(0x7fd50c1430c0 legacy=0x7fd50c1e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:49.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.050+0000 7fd5097fa640 1 -- 172.21.15.44:0/2591205847 >> v1:172.21.15.44:6789/0 conn(0x7fd50c13f710 legacy=0x7fd50c1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:49.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.051+0000 7fd5097fa640 10 monclient: _finish_hunting 0 2024-03-25T19:42:49.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.051+0000 7fd5097fa640 1 monclient: found mon.c 2024-03-25T19:42:49.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.051+0000 7fd5097fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:49.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.051+0000 7fd5097fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:42:49.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.051+0000 7fd5097fa640 1 -- 172.21.15.44:0/2591205847 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50c131980 con 0x7fd50c0a98f0 2024-03-25T19:42:49.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.051+0000 7fd5097fa640 10 monclient: _finish_auth 0 2024-03-25T19:42:49.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.051+0000 7fd5097fa640 10 monclient: _check_auth_tickets 2024-03-25T19:42:49.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.051+0000 7fd5097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:49.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.051+0000 7fd512d89640 5 monclient: authenticate success, global_id 4544 2024-03-25T19:42:49.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.051+0000 7fd512d89640 10 monclient: _renew_subs 2024-03-25T19:42:49.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.051+0000 7fd512d89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:42:49.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.051+0000 7fd512d89640 1 -- 172.21.15.44:0/2591205847 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd50c135c60 con 0x7fd50c0a98f0 2024-03-25T19:42:49.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.051+0000 7fd512d89640 10 client.4544.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:49.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.051+0000 7fd512d89640 10 monclient: _renew_subs 2024-03-25T19:42:49.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.051+0000 7fd512d89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:42:49.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.051+0000 7fd512d89640 1 -- 172.21.15.44:0/2591205847 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd50c0b7770 con 0x7fd50c0a98f0 2024-03-25T19:42:49.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.051+0000 7fd512d89640 1 librados: init done 2024-03-25T19:42:49.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.052+0000 7fd512d89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:49.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.052+0000 7fd512d89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:49.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.052+0000 7fd512d89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:42:49.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.052+0000 7fd512d89640 1 -- 172.21.15.44:0/2591205847 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd50c1371d0 con 0x7fd50c0a98f0 2024-03-25T19:42:49.384 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.383+0000 7fd5097fa640 1 -- 172.21.15.44:0/2591205847 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd500001c70 con 0x7fd50c0a98f0 2024-03-25T19:42:49.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.383+0000 7fd5097fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:49.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.383+0000 7fd5097fa640 1 -- 172.21.15.44:0/2591205847 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fd500003a60 con 0x7fd50c0a98f0 2024-03-25T19:42:49.416 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.383+0000 7fd5097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:49.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.383+0000 7fd5097fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:49.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.383+0000 7fd5097fa640 10 monclient: dump: 2024-03-25T19:42:49.417 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:49.417 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:49.417 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:49.417 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:49.417 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:49.417 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:49.417 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:49.417 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:49.417 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:49.417 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:49.417 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.383+0000 7fd50affd640 4 set_mon_vals no callback set 2024-03-25T19:42:49.418 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.384+0000 7fd5097fa640 1 -- 172.21.15.44:0/2591205847 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7fd500003c80 con 0x7fd50c0a98f0 2024-03-25T19:42:49.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.384+0000 7fd50bfff640 10 monclient: get_auth_request con 0x7fd4f006e040 auth_method 0 2024-03-25T19:42:49.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.385+0000 7fd5097fa640 10 client.4544.objecter ms_handle_connect 0x7fd4f006e040 2024-03-25T19:42:49.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.391+0000 7fd5097fa640 1 -- 172.21.15.44:0/2591205847 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7fd500054830 con 0x7fd50c0a98f0 2024-03-25T19:42:49.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.391+0000 7fd5097fa640 10 client.4544.objecter ms_dispatch 0x7fd50c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:49.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.391+0000 7fd5097fa640 3 client.4544.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:49.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.391+0000 7fd5097fa640 3 client.4544.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:49.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.391+0000 7fd5097fa640 20 client.4544.objecter dump_active .. 0 homeless 2024-03-25T19:42:49.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.391+0000 7fd5097fa640 1 -- 172.21.15.44:0/2591205847 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd500050340 con 0x7fd50c0a98f0 2024-03-25T19:42:49.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.391+0000 7fd5097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:49.419 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.391+0000 7fd5097fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:49.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.630+0000 7fd512d89640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:49.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.630+0000 7fd512d89640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:49.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.630+0000 7fd512d89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:42:49.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.630+0000 7fd512d89640 1 -- 172.21.15.44:0/2591205847 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd50c074cc0 con 0x7fd50c0a98f0 2024-03-25T19:42:49.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.666+0000 7fd5097fa640 1 -- 172.21.15.44:0/2591205847 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7fd500052270 con 0x7fd50c0a98f0 2024-03-25T19:42:49.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.666+0000 7fd5097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:49.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.666+0000 7fd5097fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:49.666 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:42:49.666 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:49.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.668+0000 7fd512d89640 10 librados: watch_flush enter 2024-03-25T19:42:49.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.668+0000 7fd512d89640 10 librados: watch_flush exit 2024-03-25T19:42:49.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.668+0000 7fd512d89640 20 client.4544.objecter shutdown clearing up homeless session... 2024-03-25T19:42:49.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.668+0000 7fd512d89640 10 client.4544.objecter successfully canceled tick 2024-03-25T19:42:49.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.668+0000 7fd512d89640 1 -- 172.21.15.44:0/2591205847 >> v1:172.21.15.111:6800/3883215793 conn(0x7fd4f006e040 legacy=0x7fd4f0070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:49.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.668+0000 7fd512d89640 10 monclient: shutdown 2024-03-25T19:42:49.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.668+0000 7fd512d89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:49.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.668+0000 7fd512d89640 1 -- 172.21.15.44:0/2591205847 >> v1:172.21.15.44:6790/0 conn(0x7fd50c0a98f0 legacy=0x7fd50c130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:49.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.668+0000 7fd512d89640 1 -- 172.21.15.44:0/2591205847 shutdown_connections 2024-03-25T19:42:49.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.668+0000 7fd512d89640 1 -- 172.21.15.44:0/2591205847 >> 172.21.15.44:0/2591205847 conn(0x7fd50c0b7ea0 msgr2=0x7fd50c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:49.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.668+0000 7fd512d89640 1 -- 172.21.15.44:0/2591205847 shutdown_connections 2024-03-25T19:42:49.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.668+0000 7fd512d89640 1 -- 172.21.15.44:0/2591205847 wait complete. 2024-03-25T19:42:49.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:49.668+0000 7fd512d89640 1 librados: shutdown 2024-03-25T19:42:49.681 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:56.488 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:56.601 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.601+0000 7f67523bc640 10 monclient: get_monmap_and_config 2024-03-25T19:42:56.601 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.601+0000 7f67523bc640 10 monclient: build_initial_monmap 2024-03-25T19:42:56.601 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.601+0000 7f67523bc640 10 monclient: monmap: 2024-03-25T19:42:56.601 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:42:56.601 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:56.601 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:42:56.602392+0000 2024-03-25T19:42:56.601 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:42:56.602392+0000 2024-03-25T19:42:56.601 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:56.601 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:56.602 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:42:56.602 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:42:56.602 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:42:56.602 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:56.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.602+0000 7f67523bc640 1 Processor -- start 2024-03-25T19:42:56.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.602+0000 7f67523bc640 1 -- start start 2024-03-25T19:42:56.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.602+0000 7f67523bc640 10 monclient: init 2024-03-25T19:42:56.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.602+0000 7f67523bc640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:56.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.602+0000 7f67523bc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:56.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.602+0000 7f67523bc640 10 monclient(hunting): picked mon.noname-c con 0x7f674c0b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T19:42:56.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.602+0000 7f67523bc640 10 monclient(hunting): picked mon.noname-a con 0x7f674c0b1730 addr v1:172.21.15.44:6789/0 2024-03-25T19:42:56.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.602+0000 7f67523bc640 10 monclient(hunting): picked mon.noname-b con 0x7f674c14be90 addr v1:172.21.15.44:6790/0 2024-03-25T19:42:56.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.602+0000 7f67523bc640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f674c0af940 con 0x7f674c0b1730 2024-03-25T19:42:56.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.602+0000 7f67523bc640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f674c0b0a70 con 0x7f674c0b4d60 2024-03-25T19:42:56.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.602+0000 7f67523bc640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f674c0b0d50 con 0x7f674c14be90 2024-03-25T19:42:56.603 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.602+0000 7f67523bc640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:56.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.602+0000 7f67523bc640 10 monclient(hunting): authenticate will time out at 5840.773708s 2024-03-25T19:42:56.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.623+0000 7f674b7fe640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f674c0b1730 0x7f674c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:50894/0 (socket says 172.21.15.44:50894) 2024-03-25T19:42:56.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.623+0000 7f674b7fe640 1 -- 172.21.15.44:0/1524089862 learned_addr learned my addr 172.21.15.44:0/1524089862 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:42:56.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.640+0000 7f674affd640 1 -- 172.21.15.44:0/1524089862 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1814356234 0 0) 0x7f674c0b0d50 con 0x7f674c14be90 2024-03-25T19:42:56.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.640+0000 7f674affd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:56.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.640+0000 7f674affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:56.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.640+0000 7f674affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:56.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.640+0000 7f674affd640 10 monclient(hunting): my global_id is 4547 2024-03-25T19:42:56.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.640+0000 7f674affd640 1 -- 172.21.15.44:0/1524089862 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6734003520 con 0x7f674c14be90 2024-03-25T19:42:56.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.648+0000 7f674affd640 1 -- 172.21.15.44:0/1524089862 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 995322094 0 0) 0x7f674c0b0a70 con 0x7f674c0b4d60 2024-03-25T19:42:56.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.648+0000 7f674affd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:56.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.648+0000 7f674affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:56.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.648+0000 7f674affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:56.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.648+0000 7f674affd640 10 monclient(hunting): my global_id is 4558 2024-03-25T19:42:56.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.649+0000 7f674affd640 1 -- 172.21.15.44:0/1524089862 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f674c0b0d50 con 0x7f674c0b4d60 2024-03-25T19:42:56.653 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.653+0000 7f674affd640 1 -- 172.21.15.44:0/1524089862 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 354911544 0 0) 0x7f674c0af940 con 0x7f674c0b1730 2024-03-25T19:42:56.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.653+0000 7f674affd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:56.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.653+0000 7f674affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:56.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.653+0000 7f674affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:56.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.653+0000 7f674affd640 10 monclient(hunting): my global_id is 4572 2024-03-25T19:42:56.654 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.653+0000 7f674affd640 1 -- 172.21.15.44:0/1524089862 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f674c0b0a70 con 0x7f674c0b1730 2024-03-25T19:42:56.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.655+0000 7f674affd640 1 -- 172.21.15.44:0/1524089862 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3266716692 0 0) 0x7f6734003520 con 0x7f674c14be90 2024-03-25T19:42:56.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.655+0000 7f674affd640 1 -- 172.21.15.44:0/1524089862 >> v1:172.21.15.111:6789/0 conn(0x7f674c0b4d60 legacy=0x7f674c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:56.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.655+0000 7f674affd640 1 -- 172.21.15.44:0/1524089862 >> v1:172.21.15.44:6789/0 conn(0x7f674c0b1730 legacy=0x7f674c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:56.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.655+0000 7f674affd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:56.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.655+0000 7f674affd640 1 monclient: found mon.noname-b 2024-03-25T19:42:56.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.655+0000 7f674affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:56.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.655+0000 7f674affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:42:56.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.655+0000 7f674affd640 1 -- 172.21.15.44:0/1524089862 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674c0b1030 con 0x7f674c14be90 2024-03-25T19:42:56.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.655+0000 7f674affd640 10 monclient: _finish_auth 0 2024-03-25T19:42:56.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.655+0000 7f674affd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:56.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.655+0000 7f674affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:56.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.655+0000 7f674affd640 1 -- 172.21.15.44:0/1524089862 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6744001a10 con 0x7f674c14be90 2024-03-25T19:42:56.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.655+0000 7f67523bc640 5 monclient: authenticate success, global_id 4547 2024-03-25T19:42:56.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.655+0000 7f67523bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:56.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.655+0000 7f674affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:56.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.655+0000 7f674affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:56.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.655+0000 7f674affd640 10 monclient: dump: 2024-03-25T19:42:56.656 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:56.657 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:56.657 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:56.657 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:56.657 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:56.657 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:56.657 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:56.657 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:56.657 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:56.657 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:56.657 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.655+0000 7f67523bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:56.674 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.674+0000 7f674affd640 1 -- 172.21.15.44:0/1524089862 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6744001b70 con 0x7f674c14be90 2024-03-25T19:42:56.674 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.674+0000 7f674affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:56.674 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.674+0000 7f674affd640 1 -- 172.21.15.44:0/1524089862 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6744001f80 con 0x7f674c14be90 2024-03-25T19:42:56.674 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.674+0000 7f67523bc640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:56.674 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.674+0000 7f67523bc640 4 set_mon_vals no callback set 2024-03-25T19:42:56.674 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.674+0000 7f674affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:56.674 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.674+0000 7f674affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:56.674 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.674+0000 7f674affd640 10 monclient: dump: 2024-03-25T19:42:56.674 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:56.674 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:56.674 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:56.675 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:56.675 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:56.675 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:56.675 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:56.675 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:56.675 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:56.675 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:56.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.674+0000 7f67523bc640 10 monclient: shutdown 2024-03-25T19:42:56.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.674+0000 7f67523bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:56.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.674+0000 7f67523bc640 1 -- 172.21.15.44:0/1524089862 >> v1:172.21.15.44:6790/0 conn(0x7f674c14be90 legacy=0x7f674c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:56.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.674+0000 7f67523bc640 1 -- 172.21.15.44:0/1524089862 shutdown_connections 2024-03-25T19:42:56.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.674+0000 7f67523bc640 1 -- 172.21.15.44:0/1524089862 >> 172.21.15.44:0/1524089862 conn(0x7f674c0617a0 msgr2=0x7f674c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:56.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.674+0000 7f67523bc640 1 -- 172.21.15.44:0/1524089862 shutdown_connections 2024-03-25T19:42:56.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.674+0000 7f67523bc640 1 -- 172.21.15.44:0/1524089862 wait complete. 2024-03-25T19:42:56.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.675+0000 7f67523bc640 10 monclient: build_initial_monmap 2024-03-25T19:42:56.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.675+0000 7f67523bc640 10 monclient: monmap: 2024-03-25T19:42:56.678 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:42:56.678 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:56.678 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:42:56.678 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:42:56.678 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:56.678 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:42:56.678 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:42:56.678 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:42:56.678 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:42:56.678 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:56.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.675+0000 7f67523bc640 1 librados: starting msgr at 2024-03-25T19:42:56.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.675+0000 7f67523bc640 1 librados: starting objecter 2024-03-25T19:42:56.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.676+0000 7f67523bc640 1 Processor -- start 2024-03-25T19:42:56.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.676+0000 7f67523bc640 1 -- start start 2024-03-25T19:42:56.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.676+0000 7f67523bc640 1 librados: setting wanted keys 2024-03-25T19:42:56.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.676+0000 7f67523bc640 1 librados: calling monclient init 2024-03-25T19:42:56.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.676+0000 7f67523bc640 10 monclient: init 2024-03-25T19:42:56.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.676+0000 7f67523bc640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:56.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.676+0000 7f67523bc640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:56.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.676+0000 7f67523bc640 10 monclient(hunting): picked mon.noname-c con 0x7f674c0b1730 addr v1:172.21.15.44:6790/0 2024-03-25T19:42:56.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.676+0000 7f67523bc640 10 monclient(hunting): picked mon.noname-a con 0x7f674c0b4d60 addr v1:172.21.15.44:6789/0 2024-03-25T19:42:56.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.676+0000 7f67523bc640 10 monclient(hunting): picked mon.noname-b con 0x7f674c14be90 addr v1:172.21.15.111:6789/0 2024-03-25T19:42:56.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.676+0000 7f67523bc640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f674c1408b0 con 0x7f674c0b4d60 2024-03-25T19:42:56.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.676+0000 7f67523bc640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f674c138370 con 0x7f674c14be90 2024-03-25T19:42:56.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.676+0000 7f67523bc640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f674c138500 con 0x7f674c0b1730 2024-03-25T19:42:56.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.676+0000 7f67523bc640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:56.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.676+0000 7f67523bc640 10 monclient(hunting): authenticate will time out at 5840.847796s 2024-03-25T19:42:56.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.688+0000 7f674bfff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f674c0b1730 0x7f674c131f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:40056/0 (socket says 172.21.15.44:40056) 2024-03-25T19:42:56.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.688+0000 7f674bfff640 1 -- 172.21.15.44:0/3644412457 learned_addr learned my addr 172.21.15.44:0/3644412457 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:42:56.689 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.690+0000 7f6748ff9640 10 client.?.objecter ms_handle_connect 0x7f674c0b1730 2024-03-25T19:42:56.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.690+0000 7f6748ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:56.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.698+0000 7f6748ff9640 10 client.?.objecter ms_handle_connect 0x7f674c0b4d60 2024-03-25T19:42:56.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.698+0000 7f6748ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:56.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.702+0000 7f6748ff9640 1 -- 172.21.15.44:0/3644412457 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1577958208 0 0) 0x7f674c138500 con 0x7f674c0b1730 2024-03-25T19:42:56.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.702+0000 7f6748ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:56.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.702+0000 7f6748ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:56.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.702+0000 7f6748ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:56.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.702+0000 7f6748ff9640 10 monclient(hunting): my global_id is 4550 2024-03-25T19:42:56.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.702+0000 7f6748ff9640 1 -- 172.21.15.44:0/3644412457 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6738004eb0 con 0x7f674c0b1730 2024-03-25T19:42:56.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.704+0000 7f6748ff9640 1 -- 172.21.15.44:0/3644412457 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1961765482 0 0) 0x7f674c1408b0 con 0x7f674c0b4d60 2024-03-25T19:42:56.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.704+0000 7f6748ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:56.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.704+0000 7f6748ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:56.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.704+0000 7f6748ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:56.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.704+0000 7f6748ff9640 10 monclient(hunting): my global_id is 4575 2024-03-25T19:42:56.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.705+0000 7f6748ff9640 1 -- 172.21.15.44:0/3644412457 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f674c138500 con 0x7f674c0b4d60 2024-03-25T19:42:56.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.707+0000 7f6748ff9640 10 client.?.objecter ms_handle_connect 0x7f674c14be90 2024-03-25T19:42:56.707 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.707+0000 7f6748ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:56.722 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.721+0000 7f6748ff9640 1 -- 172.21.15.44:0/3644412457 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2427518864 0 0) 0x7f6738004eb0 con 0x7f674c0b1730 2024-03-25T19:42:56.722 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.721+0000 7f6748ff9640 1 -- 172.21.15.44:0/3644412457 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f674c1408b0 con 0x7f674c0b1730 2024-03-25T19:42:56.722 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.721+0000 7f6748ff9640 1 -- 172.21.15.44:0/3644412457 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f673c001a30 con 0x7f674c0b1730 2024-03-25T19:42:56.722 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.721+0000 7f6748ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:56.722 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.722+0000 7f6748ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:56.722 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.722+0000 7f6748ff9640 10 monclient(hunting): dump: 2024-03-25T19:42:56.722 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:56.722 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:56.722 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:56.722 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:56.722 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:56.722 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:56.722 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:56.723 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:56.723 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:56.723 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:56.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.722+0000 7f6748ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:56.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.728+0000 7f6748ff9640 1 -- 172.21.15.44:0/3644412457 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2718730854 0 0) 0x7f674c1408b0 con 0x7f674c0b1730 2024-03-25T19:42:56.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.728+0000 7f6748ff9640 1 -- 172.21.15.44:0/3644412457 >> v1:172.21.15.111:6789/0 conn(0x7f674c14be90 legacy=0x7f674c142e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:56.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.728+0000 7f6748ff9640 1 -- 172.21.15.44:0/3644412457 >> v1:172.21.15.44:6789/0 conn(0x7f674c0b4d60 legacy=0x7f674c142740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:56.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.728+0000 7f6748ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:42:56.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.728+0000 7f6748ff9640 1 monclient: found mon.c 2024-03-25T19:42:56.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.728+0000 7f6748ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:56.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.728+0000 7f6748ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:42:56.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.728+0000 7f6748ff9640 1 -- 172.21.15.44:0/3644412457 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674c140380 con 0x7f674c0b1730 2024-03-25T19:42:56.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.728+0000 7f6748ff9640 10 monclient: _finish_auth 0 2024-03-25T19:42:56.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.728+0000 7f6748ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:42:56.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.728+0000 7f6748ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:56.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.728+0000 7f67523bc640 5 monclient: authenticate success, global_id 4550 2024-03-25T19:42:56.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.728+0000 7f67523bc640 10 monclient: _renew_subs 2024-03-25T19:42:56.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.728+0000 7f67523bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:42:56.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.728+0000 7f67523bc640 1 -- 172.21.15.44:0/3644412457 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f674c137d20 con 0x7f674c0b1730 2024-03-25T19:42:56.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.729+0000 7f67523bc640 10 client.4550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:56.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.729+0000 7f67523bc640 10 monclient: _renew_subs 2024-03-25T19:42:56.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.729+0000 7f67523bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:42:56.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.729+0000 7f67523bc640 1 -- 172.21.15.44:0/3644412457 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f674c132800 con 0x7f674c0b1730 2024-03-25T19:42:56.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.729+0000 7f67523bc640 1 librados: init done 2024-03-25T19:42:56.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.730+0000 7f67523bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:56.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.730+0000 7f67523bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:56.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.730+0000 7f67523bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:42:56.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.730+0000 7f67523bc640 1 -- 172.21.15.44:0/3644412457 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f674c1405d0 con 0x7f674c0b1730 2024-03-25T19:42:56.947 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.947+0000 7f6748ff9640 1 -- 172.21.15.44:0/3644412457 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f673c001c70 con 0x7f674c0b1730 2024-03-25T19:42:56.947 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.947+0000 7f6748ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:56.947 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.947+0000 7f6748ff9640 1 -- 172.21.15.44:0/3644412457 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f673c003a60 con 0x7f674c0b1730 2024-03-25T19:42:56.947 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.947+0000 7f6748ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:56.947 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.947+0000 7f6748ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:56.947 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.947+0000 7f6748ff9640 10 monclient: dump: 2024-03-25T19:42:56.948 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:42:56.948 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:42:56.948 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:56.948 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:42:56.948 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:56.948 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:42:56.948 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:42:56.948 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:42:56.948 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:42:56.948 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:42:56.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:56.947+0000 7f674a7fc640 4 set_mon_vals no callback set 2024-03-25T19:42:57.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.051+0000 7f6748ff9640 1 -- 172.21.15.44:0/3644412457 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f673c003bc0 con 0x7f674c0b1730 2024-03-25T19:42:57.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.052+0000 7f674b7fe640 10 monclient: get_auth_request con 0x7f673806dd10 auth_method 0 2024-03-25T19:42:57.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.052+0000 7f6748ff9640 10 client.4550.objecter ms_handle_connect 0x7f673806dd10 2024-03-25T19:42:57.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.057+0000 7f6748ff9640 1 -- 172.21.15.44:0/3644412457 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f673c0548f0 con 0x7f674c0b1730 2024-03-25T19:42:57.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.057+0000 7f6748ff9640 10 client.4550.objecter ms_dispatch 0x7f674c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:57.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.057+0000 7f6748ff9640 3 client.4550.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:57.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.057+0000 7f6748ff9640 3 client.4550.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:57.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.057+0000 7f6748ff9640 20 client.4550.objecter dump_active .. 0 homeless 2024-03-25T19:42:57.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.057+0000 7f6748ff9640 1 -- 172.21.15.44:0/3644412457 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f673c050400 con 0x7f674c0b1730 2024-03-25T19:42:57.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.057+0000 7f6748ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:57.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.057+0000 7f6748ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:57.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.322+0000 7f67523bc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:57.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.322+0000 7f67523bc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:57.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.322+0000 7f67523bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:42:57.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.322+0000 7f67523bc640 1 -- 172.21.15.44:0/3644412457 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f674c0627b0 con 0x7f674c0b1730 2024-03-25T19:42:57.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.365+0000 7f6748ff9640 1 -- 172.21.15.44:0/3644412457 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f673c052330 con 0x7f674c0b1730 2024-03-25T19:42:57.366 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.365+0000 7f6748ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:57.366 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.365+0000 7f6748ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:57.366 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:42:57.366 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:57.368 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.367+0000 7f67523bc640 10 librados: watch_flush enter 2024-03-25T19:42:57.368 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.367+0000 7f67523bc640 10 librados: watch_flush exit 2024-03-25T19:42:57.368 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.367+0000 7f67523bc640 20 client.4550.objecter shutdown clearing up homeless session... 2024-03-25T19:42:57.368 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.367+0000 7f67523bc640 10 client.4550.objecter successfully canceled tick 2024-03-25T19:42:57.368 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.367+0000 7f67523bc640 1 -- 172.21.15.44:0/3644412457 >> v1:172.21.15.111:6800/3883215793 conn(0x7f673806dd10 legacy=0x7f67380701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:57.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.367+0000 7f67523bc640 10 monclient: shutdown 2024-03-25T19:42:57.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.367+0000 7f67523bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:57.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.367+0000 7f67523bc640 1 -- 172.21.15.44:0/3644412457 >> v1:172.21.15.44:6790/0 conn(0x7f674c0b1730 legacy=0x7f674c131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:57.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.367+0000 7f67523bc640 1 -- 172.21.15.44:0/3644412457 shutdown_connections 2024-03-25T19:42:57.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.367+0000 7f67523bc640 1 -- 172.21.15.44:0/3644412457 >> 172.21.15.44:0/3644412457 conn(0x7f674c0617a0 msgr2=0x7f674c13c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:57.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.367+0000 7f67523bc640 1 -- 172.21.15.44:0/3644412457 shutdown_connections 2024-03-25T19:42:57.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.367+0000 7f67523bc640 1 -- 172.21.15.44:0/3644412457 wait complete. 2024-03-25T19:42:57.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:42:57.368+0000 7f67523bc640 1 librados: shutdown 2024-03-25T19:42:57.380 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:00.046 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:07:50.948 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:00.054 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:07:50.956 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:00.096 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:07:50.976 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:00.100 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:07:51.002 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:00.108 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:07:50.988 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:00.109 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:07:51.011 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:00.113 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:07:51.014 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:00.135 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:07:51.036 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:00.150 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:07:51.030 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:04.186 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:04.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.304+0000 7efc50af6640 10 monclient: get_monmap_and_config 2024-03-25T19:43:04.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.304+0000 7efc50af6640 10 monclient: build_initial_monmap 2024-03-25T19:43:04.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.304+0000 7efc50af6640 10 monclient: monmap: 2024-03-25T19:43:04.304 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:43:04.304 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:04.304 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:43:04.305313+0000 2024-03-25T19:43:04.304 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:43:04.305313+0000 2024-03-25T19:43:04.304 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:04.304 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:04.305 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:43:04.305 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:43:04.305 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:43:04.305 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:04.306 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.304+0000 7efc50af6640 1 Processor -- start 2024-03-25T19:43:04.306 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.304+0000 7efc50af6640 1 -- start start 2024-03-25T19:43:04.306 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.304+0000 7efc50af6640 10 monclient: init 2024-03-25T19:43:04.306 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.305+0000 7efc50af6640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:04.306 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.305+0000 7efc50af6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:04.306 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.305+0000 7efc50af6640 10 monclient(hunting): picked mon.noname-c con 0x7efc4c0a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:43:04.306 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.306+0000 7efc50af6640 10 monclient(hunting): picked mon.noname-a con 0x7efc4c13f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:43:04.306 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.306+0000 7efc50af6640 10 monclient(hunting): picked mon.noname-b con 0x7efc4c1430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:43:04.306 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.306+0000 7efc50af6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc4c142520 con 0x7efc4c13f710 2024-03-25T19:43:04.306 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.306+0000 7efc50af6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc4c0b28a0 con 0x7efc4c0a98f0 2024-03-25T19:43:04.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.306+0000 7efc50af6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc4c0b2a30 con 0x7efc4c1430c0 2024-03-25T19:43:04.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.306+0000 7efc50af6640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:04.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.306+0000 7efc50af6640 10 monclient(hunting): authenticate will time out at 5848.476964s 2024-03-25T19:43:04.314 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.313+0000 7efc4a59c640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7efc4c0a98f0 0x7efc4c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:48838/0 (socket says 172.21.15.44:48838) 2024-03-25T19:43:04.314 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.314+0000 7efc4a59c640 1 -- 172.21.15.44:0/1418271917 learned_addr learned my addr 172.21.15.44:0/1418271917 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:43:04.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.323+0000 7efc4959a640 1 -- 172.21.15.44:0/1418271917 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318352961 0 0) 0x7efc4c142520 con 0x7efc4c13f710 2024-03-25T19:43:04.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.323+0000 7efc4959a640 10 monclient(hunting): _negotiate 2024-03-25T19:43:04.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.323+0000 7efc4959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:04.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.323+0000 7efc4959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:04.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.323+0000 7efc4959a640 10 monclient(hunting): my global_id is 4578 2024-03-25T19:43:04.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.323+0000 7efc4959a640 1 -- 172.21.15.44:0/1418271917 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc34003520 con 0x7efc4c13f710 2024-03-25T19:43:04.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.323+0000 7efc4959a640 1 -- 172.21.15.44:0/1418271917 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3518155076 0 0) 0x7efc4c0b28a0 con 0x7efc4c0a98f0 2024-03-25T19:43:04.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.323+0000 7efc4959a640 10 monclient(hunting): _negotiate 2024-03-25T19:43:04.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.323+0000 7efc4959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:04.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.323+0000 7efc4959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:04.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.323+0000 7efc4959a640 10 monclient(hunting): my global_id is 4564 2024-03-25T19:43:04.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.323+0000 7efc4959a640 1 -- 172.21.15.44:0/1418271917 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc4c142520 con 0x7efc4c0a98f0 2024-03-25T19:43:04.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.329+0000 7efc4959a640 1 -- 172.21.15.44:0/1418271917 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 714703143 0 0) 0x7efc4c142520 con 0x7efc4c0a98f0 2024-03-25T19:43:04.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.329+0000 7efc4959a640 1 -- 172.21.15.44:0/1418271917 >> v1:172.21.15.44:6790/0 conn(0x7efc4c1430c0 legacy=0x7efc4c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:04.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.329+0000 7efc4959a640 1 -- 172.21.15.44:0/1418271917 >> v1:172.21.15.44:6789/0 conn(0x7efc4c13f710 legacy=0x7efc4c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:04.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.329+0000 7efc4959a640 10 monclient: _finish_hunting 0 2024-03-25T19:43:04.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.329+0000 7efc4959a640 1 monclient: found mon.noname-c 2024-03-25T19:43:04.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.329+0000 7efc4959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:04.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.329+0000 7efc4959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:43:04.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.329+0000 7efc4959a640 1 -- 172.21.15.44:0/1418271917 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc4c0b44c0 con 0x7efc4c0a98f0 2024-03-25T19:43:04.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.329+0000 7efc4959a640 10 monclient: _finish_auth 0 2024-03-25T19:43:04.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.329+0000 7efc4959a640 10 monclient: _check_auth_tickets 2024-03-25T19:43:04.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.329+0000 7efc4959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:04.331 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.329+0000 7efc50af6640 5 monclient: authenticate success, global_id 4564 2024-03-25T19:43:04.331 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.329+0000 7efc4959a640 1 -- 172.21.15.44:0/1418271917 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7efc3c001670 con 0x7efc4c0a98f0 2024-03-25T19:43:04.331 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.329+0000 7efc50af6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:04.331 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.329+0000 7efc4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:04.331 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.329+0000 7efc4959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:04.331 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.329+0000 7efc4959a640 10 monclient: dump: 2024-03-25T19:43:04.331 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:04.331 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:04.331 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:04.331 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:04.331 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:04.331 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:04.331 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:04.331 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:04.331 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:04.332 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:04.332 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.329+0000 7efc50af6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:04.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.339+0000 7efc4959a640 1 -- 172.21.15.44:0/1418271917 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc3c0017d0 con 0x7efc4c0a98f0 2024-03-25T19:43:04.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.339+0000 7efc4959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:04.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.339+0000 7efc4959a640 1 -- 172.21.15.44:0/1418271917 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7efc3c001ba0 con 0x7efc4c0a98f0 2024-03-25T19:43:04.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.339+0000 7efc50af6640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:04.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.339+0000 7efc50af6640 4 set_mon_vals no callback set 2024-03-25T19:43:04.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.339+0000 7efc4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:04.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.339+0000 7efc4959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:04.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.339+0000 7efc4959a640 10 monclient: dump: 2024-03-25T19:43:04.340 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:04.340 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:04.340 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:04.340 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:04.340 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:04.340 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:04.340 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:04.340 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:04.341 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:04.341 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:04.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.340+0000 7efc50af6640 10 monclient: shutdown 2024-03-25T19:43:04.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.340+0000 7efc50af6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:04.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.340+0000 7efc50af6640 1 -- 172.21.15.44:0/1418271917 >> v1:172.21.15.111:6789/0 conn(0x7efc4c0a98f0 legacy=0x7efc4c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:04.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.340+0000 7efc50af6640 1 -- 172.21.15.44:0/1418271917 shutdown_connections 2024-03-25T19:43:04.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.340+0000 7efc50af6640 1 -- 172.21.15.44:0/1418271917 >> 172.21.15.44:0/1418271917 conn(0x7efc4c0b7ea0 msgr2=0x7efc4c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:04.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.340+0000 7efc50af6640 1 -- 172.21.15.44:0/1418271917 shutdown_connections 2024-03-25T19:43:04.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.340+0000 7efc50af6640 1 -- 172.21.15.44:0/1418271917 wait complete. 2024-03-25T19:43:04.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.341+0000 7efc50af6640 10 monclient: build_initial_monmap 2024-03-25T19:43:04.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.341+0000 7efc50af6640 10 monclient: monmap: 2024-03-25T19:43:04.343 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:43:04.343 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:04.343 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:43:04.343 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:43:04.343 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:04.344 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:43:04.344 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:43:04.344 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:43:04.344 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:43:04.344 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:04.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.342+0000 7efc50af6640 1 librados: starting msgr at 2024-03-25T19:43:04.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.342+0000 7efc50af6640 1 librados: starting objecter 2024-03-25T19:43:04.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.342+0000 7efc50af6640 1 Processor -- start 2024-03-25T19:43:04.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.342+0000 7efc50af6640 1 -- start start 2024-03-25T19:43:04.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.342+0000 7efc50af6640 1 librados: setting wanted keys 2024-03-25T19:43:04.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.342+0000 7efc50af6640 1 librados: calling monclient init 2024-03-25T19:43:04.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.342+0000 7efc50af6640 10 monclient: init 2024-03-25T19:43:04.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.342+0000 7efc50af6640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:04.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.342+0000 7efc50af6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:43:04.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.342+0000 7efc50af6640 10 monclient(hunting): picked mon.noname-a con 0x7efc4c0a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:43:04.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.342+0000 7efc50af6640 10 monclient(hunting): picked mon.noname-c con 0x7efc4c13f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:43:04.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.342+0000 7efc50af6640 10 monclient(hunting): picked mon.noname-b con 0x7efc4c1430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:43:04.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.342+0000 7efc50af6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc4c139d20 con 0x7efc4c0a98f0 2024-03-25T19:43:04.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.342+0000 7efc50af6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc4c13b180 con 0x7efc4c1430c0 2024-03-25T19:43:04.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.342+0000 7efc50af6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc4c14e580 con 0x7efc4c13f710 2024-03-25T19:43:04.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.342+0000 7efc50af6640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:04.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.342+0000 7efc50af6640 10 monclient(hunting): authenticate will time out at 5848.513634s 2024-03-25T19:43:04.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.350+0000 7efc4ad9d640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7efc4c1430c0 0x7efc4c14de70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:48864/0 (socket says 172.21.15.44:48864) 2024-03-25T19:43:04.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.350+0000 7efc4ad9d640 1 -- 172.21.15.44:0/3125695034 learned_addr learned my addr 172.21.15.44:0/3125695034 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:43:04.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.350+0000 7efc3affd640 10 client.?.objecter ms_handle_connect 0x7efc4c1430c0 2024-03-25T19:43:04.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.350+0000 7efc3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:04.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.354+0000 7efc3affd640 10 client.?.objecter ms_handle_connect 0x7efc4c13f710 2024-03-25T19:43:04.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.354+0000 7efc3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:04.422 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.422+0000 7efc3affd640 10 client.?.objecter ms_handle_connect 0x7efc4c0a98f0 2024-03-25T19:43:04.423 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.422+0000 7efc3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:04.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.432+0000 7efc3affd640 1 -- 172.21.15.44:0/3125695034 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4145958025 0 0) 0x7efc4c139d20 con 0x7efc4c0a98f0 2024-03-25T19:43:04.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.432+0000 7efc3affd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:04.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.432+0000 7efc3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:04.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.432+0000 7efc3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:04.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.432+0000 7efc3affd640 10 monclient(hunting): my global_id is 4581 2024-03-25T19:43:04.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.432+0000 7efc3affd640 1 -- 172.21.15.44:0/3125695034 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc2c004eb0 con 0x7efc4c0a98f0 2024-03-25T19:43:04.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.443+0000 7efc3affd640 1 -- 172.21.15.44:0/3125695034 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2171026060 0 0) 0x7efc2c004eb0 con 0x7efc4c0a98f0 2024-03-25T19:43:04.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.443+0000 7efc3affd640 1 -- 172.21.15.44:0/3125695034 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc4c139d20 con 0x7efc4c0a98f0 2024-03-25T19:43:04.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.443+0000 7efc3affd640 1 -- 172.21.15.44:0/3125695034 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7efc3c001530 con 0x7efc4c0a98f0 2024-03-25T19:43:04.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.443+0000 7efc3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:04.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.443+0000 7efc3affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:04.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.443+0000 7efc3affd640 10 monclient(hunting): dump: 2024-03-25T19:43:04.444 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:04.444 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:04.444 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:04.444 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:04.444 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:04.444 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:04.444 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:04.444 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:04.444 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:04.445 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:04.445 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.443+0000 7efc3affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:04.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.456+0000 7efc3affd640 1 -- 172.21.15.44:0/3125695034 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 115478936 0 0) 0x7efc4c139d20 con 0x7efc4c0a98f0 2024-03-25T19:43:04.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.456+0000 7efc3affd640 1 -- 172.21.15.44:0/3125695034 >> v1:172.21.15.44:6790/0 conn(0x7efc4c13f710 legacy=0x7efc4c14d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:04.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.456+0000 7efc3affd640 1 -- 172.21.15.44:0/3125695034 >> v1:172.21.15.111:6789/0 conn(0x7efc4c1430c0 legacy=0x7efc4c14de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:04.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.456+0000 7efc3affd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:04.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.456+0000 7efc3affd640 1 monclient: found mon.a 2024-03-25T19:43:04.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.456+0000 7efc3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:04.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.456+0000 7efc3affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:43:04.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.456+0000 7efc3affd640 1 -- 172.21.15.44:0/3125695034 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc4c13b3f0 con 0x7efc4c0a98f0 2024-03-25T19:43:04.457 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.456+0000 7efc3affd640 10 monclient: _finish_auth 0 2024-03-25T19:43:04.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.456+0000 7efc3affd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:04.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.456+0000 7efc3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:04.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.456+0000 7efc50af6640 5 monclient: authenticate success, global_id 4581 2024-03-25T19:43:04.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.456+0000 7efc50af6640 10 monclient: _renew_subs 2024-03-25T19:43:04.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.456+0000 7efc50af6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:43:04.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.456+0000 7efc50af6640 1 -- 172.21.15.44:0/3125695034 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc4c13b6d0 con 0x7efc4c0a98f0 2024-03-25T19:43:04.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.456+0000 7efc50af6640 10 client.4581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:04.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.456+0000 7efc50af6640 10 monclient: _renew_subs 2024-03-25T19:43:04.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.456+0000 7efc50af6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:43:04.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.456+0000 7efc50af6640 1 -- 172.21.15.44:0/3125695034 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc4c13c200 con 0x7efc4c0a98f0 2024-03-25T19:43:04.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.456+0000 7efc50af6640 1 librados: init done 2024-03-25T19:43:04.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.458+0000 7efc50af6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:04.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.458+0000 7efc50af6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:04.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.458+0000 7efc50af6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:43:04.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.458+0000 7efc50af6640 1 -- 172.21.15.44:0/3125695034 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc4c139ed0 con 0x7efc4c0a98f0 2024-03-25T19:43:04.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.774+0000 7efc3affd640 1 -- 172.21.15.44:0/3125695034 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc3c004a90 con 0x7efc4c0a98f0 2024-03-25T19:43:04.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.774+0000 7efc3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:04.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.774+0000 7efc3affd640 1 -- 172.21.15.44:0/3125695034 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7efc3c004da0 con 0x7efc4c0a98f0 2024-03-25T19:43:04.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.774+0000 7efc3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:04.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.774+0000 7efc3affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:04.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.774+0000 7efc3affd640 10 monclient: dump: 2024-03-25T19:43:04.775 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:04.775 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:04.775 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:04.776 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:04.776 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:04.776 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:04.776 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:04.776 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:04.776 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:04.776 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:04.776 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.774+0000 7efc48d99640 4 set_mon_vals no callback set 2024-03-25T19:43:04.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.775+0000 7efc3affd640 1 -- 172.21.15.44:0/3125695034 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7efc3c01ad70 con 0x7efc4c0a98f0 2024-03-25T19:43:04.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.776+0000 7efc49d9b640 10 monclient: get_auth_request con 0x7efc2c06dbe0 auth_method 0 2024-03-25T19:43:04.777 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.776+0000 7efc3affd640 10 client.4581.objecter ms_handle_connect 0x7efc2c06dbe0 2024-03-25T19:43:04.784 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.784+0000 7efc3affd640 1 -- 172.21.15.44:0/3125695034 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7efc3c054940 con 0x7efc4c0a98f0 2024-03-25T19:43:04.784 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.784+0000 7efc3affd640 10 client.4581.objecter ms_dispatch 0x7efc4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:04.784 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.784+0000 7efc3affd640 3 client.4581.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:04.784 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.784+0000 7efc3affd640 3 client.4581.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:04.784 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.784+0000 7efc3affd640 20 client.4581.objecter dump_active .. 0 homeless 2024-03-25T19:43:04.784 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.784+0000 7efc3affd640 1 -- 172.21.15.44:0/3125695034 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc3c050280 con 0x7efc4c0a98f0 2024-03-25T19:43:04.784 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.784+0000 7efc3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:04.784 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:04.784+0000 7efc3affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:05.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:05.044+0000 7efc50af6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:05.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:05.044+0000 7efc50af6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:05.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:05.044+0000 7efc50af6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:43:05.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:05.044+0000 7efc50af6640 1 -- 172.21.15.44:0/3125695034 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc4c0627b0 con 0x7efc4c0a98f0 2024-03-25T19:43:05.069 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:05.069+0000 7efc3affd640 1 -- 172.21.15.44:0/3125695034 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7efc3c052380 con 0x7efc4c0a98f0 2024-03-25T19:43:05.069 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:05.069+0000 7efc3affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:05.069 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:05.069+0000 7efc3affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:05.069 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:43:05.069 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:05.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:05.071+0000 7efc50af6640 10 librados: watch_flush enter 2024-03-25T19:43:05.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:05.071+0000 7efc50af6640 10 librados: watch_flush exit 2024-03-25T19:43:05.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:05.071+0000 7efc50af6640 20 client.4581.objecter shutdown clearing up homeless session... 2024-03-25T19:43:05.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:05.071+0000 7efc50af6640 10 client.4581.objecter successfully canceled tick 2024-03-25T19:43:05.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:05.071+0000 7efc50af6640 1 -- 172.21.15.44:0/3125695034 >> v1:172.21.15.111:6800/3883215793 conn(0x7efc2c06dbe0 legacy=0x7efc2c0700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:05.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:05.071+0000 7efc50af6640 10 monclient: shutdown 2024-03-25T19:43:05.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:05.071+0000 7efc50af6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:05.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:05.071+0000 7efc50af6640 1 -- 172.21.15.44:0/3125695034 >> v1:172.21.15.44:6789/0 conn(0x7efc4c0a98f0 legacy=0x7efc4c13baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:05.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:05.071+0000 7efc50af6640 1 -- 172.21.15.44:0/3125695034 shutdown_connections 2024-03-25T19:43:05.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:05.071+0000 7efc50af6640 1 -- 172.21.15.44:0/3125695034 >> 172.21.15.44:0/3125695034 conn(0x7efc4c0b7ea0 msgr2=0x7efc4c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:05.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:05.071+0000 7efc50af6640 1 -- 172.21.15.44:0/3125695034 shutdown_connections 2024-03-25T19:43:05.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:05.071+0000 7efc50af6640 1 -- 172.21.15.44:0/3125695034 wait complete. 2024-03-25T19:43:05.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:05.071+0000 7efc50af6640 1 librados: shutdown 2024-03-25T19:43:05.083 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:11.890 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:12.007 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.007+0000 7fe9ca9ba640 10 monclient: get_monmap_and_config 2024-03-25T19:43:12.007 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.007+0000 7fe9ca9ba640 10 monclient: build_initial_monmap 2024-03-25T19:43:12.007 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.007+0000 7fe9ca9ba640 10 monclient: monmap: 2024-03-25T19:43:12.007 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:43:12.007 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:12.007 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:43:12.008263+0000 2024-03-25T19:43:12.007 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:43:12.008263+0000 2024-03-25T19:43:12.007 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:12.007 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:12.007 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:43:12.008 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:43:12.008 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:43:12.008 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:12.008 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.007+0000 7fe9ca9ba640 1 Processor -- start 2024-03-25T19:43:12.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.008+0000 7fe9ca9ba640 1 -- start start 2024-03-25T19:43:12.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.008+0000 7fe9ca9ba640 10 monclient: init 2024-03-25T19:43:12.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.008+0000 7fe9ca9ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:12.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.008+0000 7fe9ca9ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:12.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.008+0000 7fe9ca9ba640 10 monclient(hunting): picked mon.noname-b con 0x7fe9c40a96e0 addr v1:172.21.15.44:6790/0 2024-03-25T19:43:12.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.008+0000 7fe9ca9ba640 10 monclient(hunting): picked mon.noname-c con 0x7fe9c40b1b10 addr v1:172.21.15.111:6789/0 2024-03-25T19:43:12.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.008+0000 7fe9ca9ba640 10 monclient(hunting): picked mon.noname-a con 0x7fe9c4146060 addr v1:172.21.15.44:6789/0 2024-03-25T19:43:12.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.008+0000 7fe9ca9ba640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c4140040 con 0x7fe9c4146060 2024-03-25T19:43:12.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.008+0000 7fe9ca9ba640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c40b1fa0 con 0x7fe9c40b1b10 2024-03-25T19:43:12.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.008+0000 7fe9ca9ba640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c40b24c0 con 0x7fe9c40a96e0 2024-03-25T19:43:12.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.008+0000 7fe9ca9ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:12.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.008+0000 7fe9ca9ba640 10 monclient(hunting): authenticate will time out at 5856.179601s 2024-03-25T19:43:12.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.010+0000 7fe9c37fe640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe9c40b1b10 0x7fe9c4145950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:35214/0 (socket says 172.21.15.44:35214) 2024-03-25T19:43:12.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.010+0000 7fe9c37fe640 1 -- 172.21.15.44:0/2029508029 learned_addr learned my addr 172.21.15.44:0/2029508029 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:43:12.024 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.024+0000 7fe9c2ffd640 1 -- 172.21.15.44:0/2029508029 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2521076844 0 0) 0x7fe9c40b1fa0 con 0x7fe9c40b1b10 2024-03-25T19:43:12.024 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.024+0000 7fe9c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:12.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.024+0000 7fe9c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:12.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.024+0000 7fe9c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:12.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.024+0000 7fe9c2ffd640 10 monclient(hunting): my global_id is 4570 2024-03-25T19:43:12.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.024+0000 7fe9c2ffd640 1 -- 172.21.15.44:0/2029508029 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9ac003520 con 0x7fe9c40b1b10 2024-03-25T19:43:12.030 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.030+0000 7fe9c2ffd640 1 -- 172.21.15.44:0/2029508029 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3088327341 0 0) 0x7fe9ac003520 con 0x7fe9c40b1b10 2024-03-25T19:43:12.030 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.030+0000 7fe9c2ffd640 1 -- 172.21.15.44:0/2029508029 >> v1:172.21.15.44:6790/0 conn(0x7fe9c40a96e0 legacy=0x7fe9c40b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:12.030 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.030+0000 7fe9c2ffd640 1 -- 172.21.15.44:0/2029508029 >> v1:172.21.15.44:6789/0 conn(0x7fe9c4146060 legacy=0x7fe9c414dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:12.030 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.030+0000 7fe9c2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:12.030 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.030+0000 7fe9c2ffd640 1 monclient: found mon.noname-c 2024-03-25T19:43:12.030 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.030+0000 7fe9c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:12.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.030+0000 7fe9c2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:43:12.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.030+0000 7fe9c2ffd640 1 -- 172.21.15.44:0/2029508029 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c40b15b0 con 0x7fe9c40b1b10 2024-03-25T19:43:12.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.030+0000 7fe9c2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:43:12.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.030+0000 7fe9c2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:12.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.030+0000 7fe9c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:12.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.030+0000 7fe9c2ffd640 1 -- 172.21.15.44:0/2029508029 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fe9b80016e0 con 0x7fe9c40b1b10 2024-03-25T19:43:12.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.030+0000 7fe9c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:12.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.030+0000 7fe9c2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:12.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.030+0000 7fe9c2ffd640 10 monclient: dump: 2024-03-25T19:43:12.031 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:12.031 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:12.031 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:12.031 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:12.031 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:12.031 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:12.032 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:12.032 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:12.032 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:12.032 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:12.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.030+0000 7fe9ca9ba640 5 monclient: authenticate success, global_id 4570 2024-03-25T19:43:12.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.030+0000 7fe9ca9ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:12.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.033+0000 7fe9c2ffd640 1 -- 172.21.15.44:0/2029508029 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9b8001840 con 0x7fe9c40b1b10 2024-03-25T19:43:12.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.033+0000 7fe9c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:12.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.033+0000 7fe9c2ffd640 1 -- 172.21.15.44:0/2029508029 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fe9b8001c50 con 0x7fe9c40b1b10 2024-03-25T19:43:12.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.033+0000 7fe9c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:12.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.033+0000 7fe9c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:12.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.033+0000 7fe9c2ffd640 10 monclient: dump: 2024-03-25T19:43:12.034 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:12.034 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:12.034 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:12.034 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:12.034 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:12.034 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:12.034 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:12.034 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:12.034 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:12.034 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:12.035 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.033+0000 7fe9ca9ba640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:12.035 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.033+0000 7fe9ca9ba640 4 set_mon_vals no callback set 2024-03-25T19:43:12.036 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.034+0000 7fe9ca9ba640 10 monclient: shutdown 2024-03-25T19:43:12.037 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.034+0000 7fe9ca9ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:12.037 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.034+0000 7fe9ca9ba640 1 -- 172.21.15.44:0/2029508029 >> v1:172.21.15.111:6789/0 conn(0x7fe9c40b1b10 legacy=0x7fe9c4145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:12.037 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.034+0000 7fe9ca9ba640 1 -- 172.21.15.44:0/2029508029 shutdown_connections 2024-03-25T19:43:12.037 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.034+0000 7fe9ca9ba640 1 -- 172.21.15.44:0/2029508029 >> 172.21.15.44:0/2029508029 conn(0x7fe9c40b7ea0 msgr2=0x7fe9c40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:12.037 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.034+0000 7fe9ca9ba640 1 -- 172.21.15.44:0/2029508029 shutdown_connections 2024-03-25T19:43:12.037 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.034+0000 7fe9ca9ba640 1 -- 172.21.15.44:0/2029508029 wait complete. 2024-03-25T19:43:12.037 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.035+0000 7fe9ca9ba640 10 monclient: build_initial_monmap 2024-03-25T19:43:12.037 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.035+0000 7fe9ca9ba640 10 monclient: monmap: 2024-03-25T19:43:12.037 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:43:12.037 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:12.037 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:43:12.037 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:43:12.037 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:12.038 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:43:12.038 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:43:12.038 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:43:12.038 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:43:12.038 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:12.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.035+0000 7fe9ca9ba640 1 librados: starting msgr at 2024-03-25T19:43:12.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.035+0000 7fe9ca9ba640 1 librados: starting objecter 2024-03-25T19:43:12.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.035+0000 7fe9ca9ba640 1 Processor -- start 2024-03-25T19:43:12.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.036+0000 7fe9ca9ba640 1 -- start start 2024-03-25T19:43:12.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.036+0000 7fe9ca9ba640 1 librados: setting wanted keys 2024-03-25T19:43:12.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.036+0000 7fe9ca9ba640 1 librados: calling monclient init 2024-03-25T19:43:12.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.036+0000 7fe9ca9ba640 10 monclient: init 2024-03-25T19:43:12.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.036+0000 7fe9ca9ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:12.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.036+0000 7fe9ca9ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:12.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.036+0000 7fe9ca9ba640 10 monclient(hunting): picked mon.noname-c con 0x7fe9c40a96e0 addr v1:172.21.15.44:6790/0 2024-03-25T19:43:12.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.036+0000 7fe9ca9ba640 10 monclient(hunting): picked mon.noname-b con 0x7fe9c4146060 addr v1:172.21.15.111:6789/0 2024-03-25T19:43:12.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.036+0000 7fe9ca9ba640 10 monclient(hunting): picked mon.noname-a con 0x7fe9c40afee0 addr v1:172.21.15.44:6789/0 2024-03-25T19:43:12.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.036+0000 7fe9ca9ba640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c40af820 con 0x7fe9c40afee0 2024-03-25T19:43:12.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.036+0000 7fe9ca9ba640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c40afae0 con 0x7fe9c4146060 2024-03-25T19:43:12.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.036+0000 7fe9ca9ba640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c40ae6a0 con 0x7fe9c40a96e0 2024-03-25T19:43:12.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.036+0000 7fe9ca9ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:12.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.036+0000 7fe9ca9ba640 10 monclient(hunting): authenticate will time out at 5856.207627s 2024-03-25T19:43:12.045 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.045+0000 7fe9c8f57640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fe9c40afee0 0x7fe9c41d8160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:34894/0 (socket says 172.21.15.44:34894) 2024-03-25T19:43:12.045 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.045+0000 7fe9c8f57640 1 -- 172.21.15.44:0/1848370494 learned_addr learned my addr 172.21.15.44:0/1848370494 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:43:12.048 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.048+0000 7fe9c0ff9640 10 client.?.objecter ms_handle_connect 0x7fe9c40a96e0 2024-03-25T19:43:12.048 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.048+0000 7fe9c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:12.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.049+0000 7fe9c0ff9640 10 client.?.objecter ms_handle_connect 0x7fe9c40afee0 2024-03-25T19:43:12.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.049+0000 7fe9c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:12.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.058+0000 7fe9c0ff9640 1 -- 172.21.15.44:0/1848370494 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060284803 0 0) 0x7fe9c40ae6a0 con 0x7fe9c40a96e0 2024-03-25T19:43:12.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.058+0000 7fe9c0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:12.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.058+0000 7fe9c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:12.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.058+0000 7fe9c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:12.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.058+0000 7fe9c0ff9640 10 monclient(hunting): my global_id is 4562 2024-03-25T19:43:12.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.058+0000 7fe9c0ff9640 1 -- 172.21.15.44:0/1848370494 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9b0004eb0 con 0x7fe9c40a96e0 2024-03-25T19:43:12.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.058+0000 7fe9c0ff9640 10 client.?.objecter ms_handle_connect 0x7fe9c4146060 2024-03-25T19:43:12.059 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.058+0000 7fe9c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:12.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.063+0000 7fe9c0ff9640 1 -- 172.21.15.44:0/1848370494 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028536747 0 0) 0x7fe9c40afae0 con 0x7fe9c4146060 2024-03-25T19:43:12.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.063+0000 7fe9c0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:12.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.063+0000 7fe9c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:12.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.063+0000 7fe9c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:12.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.063+0000 7fe9c0ff9640 10 monclient(hunting): my global_id is 4573 2024-03-25T19:43:12.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.063+0000 7fe9c0ff9640 1 -- 172.21.15.44:0/1848370494 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9c40ae6a0 con 0x7fe9c4146060 2024-03-25T19:43:12.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.067+0000 7fe9c0ff9640 1 -- 172.21.15.44:0/1848370494 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135581645 0 0) 0x7fe9c40af820 con 0x7fe9c40afee0 2024-03-25T19:43:12.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.067+0000 7fe9c0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:12.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.067+0000 7fe9c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:12.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.067+0000 7fe9c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:12.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.067+0000 7fe9c0ff9640 10 monclient(hunting): my global_id is 4584 2024-03-25T19:43:12.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.067+0000 7fe9c0ff9640 1 -- 172.21.15.44:0/1848370494 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9c40afae0 con 0x7fe9c40afee0 2024-03-25T19:43:12.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.073+0000 7fe9c0ff9640 1 -- 172.21.15.44:0/1848370494 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3921958043 0 0) 0x7fe9c40ae6a0 con 0x7fe9c4146060 2024-03-25T19:43:12.073 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.073+0000 7fe9c0ff9640 1 -- 172.21.15.44:0/1848370494 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9c40af820 con 0x7fe9c4146060 2024-03-25T19:43:12.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.075+0000 7fe9c0ff9640 1 -- 172.21.15.44:0/1848370494 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 7102449 0 0) 0x7fe9c40afae0 con 0x7fe9c40afee0 2024-03-25T19:43:12.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.075+0000 7fe9c0ff9640 1 -- 172.21.15.44:0/1848370494 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9c40ae6a0 con 0x7fe9c40afee0 2024-03-25T19:43:12.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.075+0000 7fe9c0ff9640 1 -- 172.21.15.44:0/1848370494 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fe9bc001bd0 con 0x7fe9c40afee0 2024-03-25T19:43:12.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.075+0000 7fe9c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:12.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.075+0000 7fe9c0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:12.076 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.075+0000 7fe9c0ff9640 10 monclient(hunting): dump: 2024-03-25T19:43:12.076 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:12.076 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:12.076 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:12.076 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:12.076 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:12.076 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:12.076 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:12.076 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:12.076 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:12.076 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:12.076 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.075+0000 7fe9c0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:12.081 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.081+0000 7fe9c0ff9640 1 -- 172.21.15.44:0/1848370494 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1476168226 0 0) 0x7fe9c40ae6a0 con 0x7fe9c40afee0 2024-03-25T19:43:12.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.081+0000 7fe9c0ff9640 1 -- 172.21.15.44:0/1848370494 >> v1:172.21.15.44:6790/0 conn(0x7fe9c40a96e0 legacy=0x7fe9c40af110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:12.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.081+0000 7fe9c0ff9640 1 -- 172.21.15.44:0/1848370494 >> v1:172.21.15.111:6789/0 conn(0x7fe9c4146060 legacy=0x7fe9c41d7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:12.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.081+0000 7fe9c0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:43:12.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.081+0000 7fe9c0ff9640 1 monclient: found mon.a 2024-03-25T19:43:12.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.081+0000 7fe9c0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:12.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.081+0000 7fe9c0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:43:12.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.081+0000 7fe9c0ff9640 1 -- 172.21.15.44:0/1848370494 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c41e63d0 con 0x7fe9c40afee0 2024-03-25T19:43:12.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.081+0000 7fe9c0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:43:12.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.081+0000 7fe9c0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:43:12.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.081+0000 7fe9c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:12.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.081+0000 7fe9ca9ba640 5 monclient: authenticate success, global_id 4584 2024-03-25T19:43:12.082 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.081+0000 7fe9ca9ba640 10 monclient: _renew_subs 2024-03-25T19:43:12.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.081+0000 7fe9ca9ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:43:12.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.081+0000 7fe9ca9ba640 1 -- 172.21.15.44:0/1848370494 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c41e33a0 con 0x7fe9c40afee0 2024-03-25T19:43:12.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.081+0000 7fe9ca9ba640 10 client.4584.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:12.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.081+0000 7fe9ca9ba640 10 monclient: _renew_subs 2024-03-25T19:43:12.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.081+0000 7fe9ca9ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:43:12.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.081+0000 7fe9ca9ba640 1 -- 172.21.15.44:0/1848370494 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c41e38d0 con 0x7fe9c40afee0 2024-03-25T19:43:12.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.081+0000 7fe9ca9ba640 1 librados: init done 2024-03-25T19:43:12.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.082+0000 7fe9ca9ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:12.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.082+0000 7fe9ca9ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:12.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.082+0000 7fe9ca9ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:43:12.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.082+0000 7fe9ca9ba640 1 -- 172.21.15.44:0/1848370494 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c40a9ac0 con 0x7fe9c40afee0 2024-03-25T19:43:12.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.413+0000 7fe9c0ff9640 1 -- 172.21.15.44:0/1848370494 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9bc0020c0 con 0x7fe9c40afee0 2024-03-25T19:43:12.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.413+0000 7fe9c0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:12.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.413+0000 7fe9c0ff9640 1 -- 172.21.15.44:0/1848370494 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fe9bc003b30 con 0x7fe9c40afee0 2024-03-25T19:43:12.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.413+0000 7fe9c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:12.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.413+0000 7fe9c0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:12.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.413+0000 7fe9c0ff9640 10 monclient: dump: 2024-03-25T19:43:12.414 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:12.414 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:12.414 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:12.414 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:12.414 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:12.414 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:12.414 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:12.414 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:12.414 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:12.414 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:12.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.413+0000 7fe9c27fc640 4 set_mon_vals no callback set 2024-03-25T19:43:12.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.414+0000 7fe9c0ff9640 1 -- 172.21.15.44:0/1848370494 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7fe9bc003d50 con 0x7fe9c40afee0 2024-03-25T19:43:12.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.414+0000 7fe9c3fff640 10 monclient: get_auth_request con 0x7fe9b006e110 auth_method 0 2024-03-25T19:43:12.415 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.415+0000 7fe9c0ff9640 10 client.4584.objecter ms_handle_connect 0x7fe9b006e110 2024-03-25T19:43:12.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.421+0000 7fe9c0ff9640 1 -- 172.21.15.44:0/1848370494 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7fe9bc054a20 con 0x7fe9c40afee0 2024-03-25T19:43:12.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.421+0000 7fe9c0ff9640 10 client.4584.objecter ms_dispatch 0x7fe9c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:12.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.421+0000 7fe9c0ff9640 3 client.4584.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:12.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.421+0000 7fe9c0ff9640 3 client.4584.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:12.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.421+0000 7fe9c0ff9640 20 client.4584.objecter dump_active .. 0 homeless 2024-03-25T19:43:12.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.421+0000 7fe9c0ff9640 1 -- 172.21.15.44:0/1848370494 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9bc050360 con 0x7fe9c40afee0 2024-03-25T19:43:12.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.421+0000 7fe9c0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:12.421 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.421+0000 7fe9c0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:12.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.643+0000 7fe9ca9ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:12.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.643+0000 7fe9ca9ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:12.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.643+0000 7fe9ca9ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:43:12.644 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.643+0000 7fe9ca9ba640 1 -- 172.21.15.44:0/1848370494 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9c40afc90 con 0x7fe9c40afee0 2024-03-25T19:43:12.696 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.696+0000 7fe9c0ff9640 1 -- 172.21.15.44:0/1848370494 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7fe9c40afc90 con 0x7fe9c40afee0 2024-03-25T19:43:12.696 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.696+0000 7fe9c0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:12.696 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.696+0000 7fe9c0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:12.696 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:43:12.697 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:12.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.699+0000 7fe9ca9ba640 10 librados: watch_flush enter 2024-03-25T19:43:12.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.699+0000 7fe9ca9ba640 10 librados: watch_flush exit 2024-03-25T19:43:12.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.699+0000 7fe9ca9ba640 20 client.4584.objecter shutdown clearing up homeless session... 2024-03-25T19:43:12.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.699+0000 7fe9ca9ba640 10 client.4584.objecter successfully canceled tick 2024-03-25T19:43:12.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.699+0000 7fe9ca9ba640 1 -- 172.21.15.44:0/1848370494 >> v1:172.21.15.111:6800/3883215793 conn(0x7fe9b006e110 legacy=0x7fe9b00705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:12.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.699+0000 7fe9ca9ba640 10 monclient: shutdown 2024-03-25T19:43:12.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.699+0000 7fe9ca9ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:12.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.699+0000 7fe9ca9ba640 1 -- 172.21.15.44:0/1848370494 >> v1:172.21.15.44:6789/0 conn(0x7fe9c40afee0 legacy=0x7fe9c41d8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:12.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.699+0000 7fe9ca9ba640 1 -- 172.21.15.44:0/1848370494 shutdown_connections 2024-03-25T19:43:12.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.699+0000 7fe9ca9ba640 1 -- 172.21.15.44:0/1848370494 >> 172.21.15.44:0/1848370494 conn(0x7fe9c40b7ea0 msgr2=0x7fe9c413f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:12.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.699+0000 7fe9ca9ba640 1 -- 172.21.15.44:0/1848370494 shutdown_connections 2024-03-25T19:43:12.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.699+0000 7fe9ca9ba640 1 -- 172.21.15.44:0/1848370494 wait complete. 2024-03-25T19:43:12.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:12.699+0000 7fe9ca9ba640 1 librados: shutdown 2024-03-25T19:43:12.711 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:19.517 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:19.631 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.631+0000 7fbaa68d9640 10 monclient: get_monmap_and_config 2024-03-25T19:43:19.631 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.631+0000 7fbaa68d9640 10 monclient: build_initial_monmap 2024-03-25T19:43:19.631 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.631+0000 7fbaa68d9640 10 monclient: monmap: 2024-03-25T19:43:19.631 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:43:19.631 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:19.632 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:43:19.632420+0000 2024-03-25T19:43:19.632 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:43:19.632420+0000 2024-03-25T19:43:19.632 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:19.632 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:19.632 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:43:19.632 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:43:19.632 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:43:19.632 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:19.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.632+0000 7fbaa68d9640 1 Processor -- start 2024-03-25T19:43:19.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.632+0000 7fbaa68d9640 1 -- start start 2024-03-25T19:43:19.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.632+0000 7fbaa68d9640 10 monclient: init 2024-03-25T19:43:19.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.632+0000 7fbaa68d9640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:19.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.632+0000 7fbaa68d9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:19.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.632+0000 7fbaa68d9640 10 monclient(hunting): picked mon.noname-a con 0x7fbaa00a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:43:19.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.632+0000 7fbaa68d9640 10 monclient(hunting): picked mon.noname-c con 0x7fbaa013f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:43:19.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.632+0000 7fbaa68d9640 10 monclient(hunting): picked mon.noname-b con 0x7fbaa01430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:43:19.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.632+0000 7fbaa68d9640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa0142520 con 0x7fbaa00a98f0 2024-03-25T19:43:19.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.632+0000 7fbaa68d9640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa00b28a0 con 0x7fbaa013f710 2024-03-25T19:43:19.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.632+0000 7fbaa68d9640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa00b2a30 con 0x7fbaa01430c0 2024-03-25T19:43:19.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.632+0000 7fbaa68d9640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:19.634 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.632+0000 7fbaa68d9640 10 monclient(hunting): authenticate will time out at 5863.803760s 2024-03-25T19:43:19.639 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.639+0000 7fba9f7fe640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fbaa013f710 0x7fbaa01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:40692/0 (socket says 172.21.15.44:40692) 2024-03-25T19:43:19.639 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.639+0000 7fba9f7fe640 1 -- 172.21.15.44:0/1503230723 learned_addr learned my addr 172.21.15.44:0/1503230723 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:43:19.650 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.650+0000 7fba9effd640 1 -- 172.21.15.44:0/1503230723 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268765861 0 0) 0x7fbaa00b28a0 con 0x7fbaa013f710 2024-03-25T19:43:19.650 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.650+0000 7fba9effd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:19.650 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.650+0000 7fba9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:19.650 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.650+0000 7fba9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:19.650 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.650+0000 7fba9effd640 10 monclient(hunting): my global_id is 4576 2024-03-25T19:43:19.650 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.650+0000 7fba9effd640 1 -- 172.21.15.44:0/1503230723 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba88003520 con 0x7fbaa013f710 2024-03-25T19:43:19.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.660+0000 7fba9effd640 1 -- 172.21.15.44:0/1503230723 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 812700636 0 0) 0x7fba88003520 con 0x7fbaa013f710 2024-03-25T19:43:19.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.660+0000 7fba9effd640 1 -- 172.21.15.44:0/1503230723 >> v1:172.21.15.44:6790/0 conn(0x7fbaa01430c0 legacy=0x7fbaa014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:19.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.660+0000 7fba9effd640 1 -- 172.21.15.44:0/1503230723 >> v1:172.21.15.44:6789/0 conn(0x7fbaa00a98f0 legacy=0x7fbaa00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:19.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.660+0000 7fba9effd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:19.660 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.660+0000 7fba9effd640 1 monclient: found mon.noname-c 2024-03-25T19:43:19.661 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.660+0000 7fba9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:19.661 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.660+0000 7fba9effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:43:19.661 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.660+0000 7fba9effd640 1 -- 172.21.15.44:0/1503230723 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa00b44c0 con 0x7fbaa013f710 2024-03-25T19:43:19.661 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.660+0000 7fba9effd640 10 monclient: _finish_auth 0 2024-03-25T19:43:19.661 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.660+0000 7fba9effd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:19.661 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.660+0000 7fba9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:19.661 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.660+0000 7fba9effd640 1 -- 172.21.15.44:0/1503230723 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fba94001690 con 0x7fbaa013f710 2024-03-25T19:43:19.661 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.660+0000 7fbaa68d9640 5 monclient: authenticate success, global_id 4576 2024-03-25T19:43:19.661 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.660+0000 7fba9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:19.661 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.660+0000 7fba9effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:19.661 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.660+0000 7fba9effd640 10 monclient: dump: 2024-03-25T19:43:19.661 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:19.661 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:19.661 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:19.662 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:19.662 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:19.662 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:19.662 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:19.662 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:19.662 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:19.662 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:19.662 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.660+0000 7fbaa68d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:19.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.666+0000 7fba9effd640 1 -- 172.21.15.44:0/1503230723 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba940017f0 con 0x7fbaa013f710 2024-03-25T19:43:19.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.666+0000 7fba9effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:19.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.666+0000 7fba9effd640 1 -- 172.21.15.44:0/1503230723 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fba94001c00 con 0x7fbaa013f710 2024-03-25T19:43:19.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.666+0000 7fba9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:19.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.666+0000 7fba9effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:19.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.666+0000 7fba9effd640 10 monclient: dump: 2024-03-25T19:43:19.666 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:19.666 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:19.666 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:19.667 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:19.667 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:19.667 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:19.667 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:19.667 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:19.667 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:19.667 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:19.667 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.666+0000 7fbaa68d9640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:19.667 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.666+0000 7fbaa68d9640 4 set_mon_vals no callback set 2024-03-25T19:43:19.668 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.667+0000 7fbaa68d9640 10 monclient: shutdown 2024-03-25T19:43:19.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.667+0000 7fbaa68d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:19.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.667+0000 7fbaa68d9640 1 -- 172.21.15.44:0/1503230723 >> v1:172.21.15.111:6789/0 conn(0x7fbaa013f710 legacy=0x7fbaa01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:19.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.667+0000 7fbaa68d9640 1 -- 172.21.15.44:0/1503230723 shutdown_connections 2024-03-25T19:43:19.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.667+0000 7fbaa68d9640 1 -- 172.21.15.44:0/1503230723 >> 172.21.15.44:0/1503230723 conn(0x7fbaa00b7ea0 msgr2=0x7fbaa00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:19.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.667+0000 7fbaa68d9640 1 -- 172.21.15.44:0/1503230723 shutdown_connections 2024-03-25T19:43:19.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.667+0000 7fbaa68d9640 1 -- 172.21.15.44:0/1503230723 wait complete. 2024-03-25T19:43:19.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.668+0000 7fbaa68d9640 10 monclient: build_initial_monmap 2024-03-25T19:43:19.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.668+0000 7fbaa68d9640 10 monclient: monmap: 2024-03-25T19:43:19.669 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:43:19.669 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:19.669 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:43:19.669 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:43:19.669 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:19.670 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:43:19.670 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:43:19.670 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:43:19.670 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:43:19.670 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:19.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.668+0000 7fbaa68d9640 1 librados: starting msgr at 2024-03-25T19:43:19.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.668+0000 7fbaa68d9640 1 librados: starting objecter 2024-03-25T19:43:19.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.668+0000 7fbaa68d9640 1 Processor -- start 2024-03-25T19:43:19.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.669+0000 7fbaa68d9640 1 -- start start 2024-03-25T19:43:19.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.669+0000 7fbaa68d9640 1 librados: setting wanted keys 2024-03-25T19:43:19.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.669+0000 7fbaa68d9640 1 librados: calling monclient init 2024-03-25T19:43:19.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.669+0000 7fbaa68d9640 10 monclient: init 2024-03-25T19:43:19.671 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.669+0000 7fbaa68d9640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:19.671 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.669+0000 7fbaa68d9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:19.672 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.669+0000 7fbaa68d9640 10 monclient(hunting): picked mon.noname-c con 0x7fbaa00a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T19:43:19.672 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.670+0000 7fbaa68d9640 10 monclient(hunting): picked mon.noname-b con 0x7fbaa013f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:43:19.672 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.670+0000 7fbaa68d9640 10 monclient(hunting): picked mon.noname-a con 0x7fbaa01430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:43:19.672 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.670+0000 7fbaa68d9640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa00b1870 con 0x7fbaa01430c0 2024-03-25T19:43:19.672 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.670+0000 7fbaa68d9640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa00ac910 con 0x7fbaa013f710 2024-03-25T19:43:19.672 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.670+0000 7fbaa68d9640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa00abb00 con 0x7fbaa00a98f0 2024-03-25T19:43:19.672 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.670+0000 7fbaa68d9640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:19.672 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.670+0000 7fbaa68d9640 10 monclient(hunting): authenticate will time out at 5863.840994s 2024-03-25T19:43:19.685 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.685+0000 7fbaa4e76640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fbaa01430c0 0x7fbaa01e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41408/0 (socket says 172.21.15.44:41408) 2024-03-25T19:43:19.685 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.685+0000 7fbaa4e76640 1 -- 172.21.15.44:0/1114698165 learned_addr learned my addr 172.21.15.44:0/1114698165 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:43:19.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.686+0000 7fba9cff9640 10 client.?.objecter ms_handle_connect 0x7fbaa00a98f0 2024-03-25T19:43:19.686 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.686+0000 7fba9cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:19.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.687+0000 7fba9cff9640 10 client.?.objecter ms_handle_connect 0x7fbaa01430c0 2024-03-25T19:43:19.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.687+0000 7fba9cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:19.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.698+0000 7fba9cff9640 10 client.?.objecter ms_handle_connect 0x7fbaa013f710 2024-03-25T19:43:19.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.698+0000 7fba9cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:19.708 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.708+0000 7fba9cff9640 1 -- 172.21.15.44:0/1114698165 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501991803 0 0) 0x7fbaa00abb00 con 0x7fbaa00a98f0 2024-03-25T19:43:19.708 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.708+0000 7fba9cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:19.708 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.708+0000 7fba9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:19.708 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.708+0000 7fba9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:19.708 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.708+0000 7fba9cff9640 10 monclient(hunting): my global_id is 4568 2024-03-25T19:43:19.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.708+0000 7fba9cff9640 1 -- 172.21.15.44:0/1114698165 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba8c004eb0 con 0x7fbaa00a98f0 2024-03-25T19:43:19.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.709+0000 7fba9cff9640 1 -- 172.21.15.44:0/1114698165 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 442638259 0 0) 0x7fbaa00ac910 con 0x7fbaa013f710 2024-03-25T19:43:19.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.709+0000 7fba9cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:19.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.709+0000 7fba9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:19.710 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.709+0000 7fba9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:19.710 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.709+0000 7fba9cff9640 10 monclient(hunting): my global_id is 4579 2024-03-25T19:43:19.710 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.709+0000 7fba9cff9640 1 -- 172.21.15.44:0/1114698165 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaa00abb00 con 0x7fbaa013f710 2024-03-25T19:43:19.718 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.718+0000 7fba9cff9640 1 -- 172.21.15.44:0/1114698165 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054253578 0 0) 0x7fbaa00b2a30 con 0x7fbaa01430c0 2024-03-25T19:43:19.718 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.718+0000 7fba9cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:19.718 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.718+0000 7fba9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:19.718 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.718+0000 7fba9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:19.718 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.718+0000 7fba9cff9640 10 monclient(hunting): my global_id is 4590 2024-03-25T19:43:19.718 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.718+0000 7fba9cff9640 1 -- 172.21.15.44:0/1114698165 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaa00ac910 con 0x7fbaa01430c0 2024-03-25T19:43:19.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.730+0000 7fba9cff9640 1 -- 172.21.15.44:0/1114698165 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3593936341 0 0) 0x7fba8c004eb0 con 0x7fbaa00a98f0 2024-03-25T19:43:19.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.730+0000 7fba9cff9640 1 -- 172.21.15.44:0/1114698165 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbaa00b2a30 con 0x7fbaa00a98f0 2024-03-25T19:43:19.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.730+0000 7fba9cff9640 1 -- 172.21.15.44:0/1114698165 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fba90001930 con 0x7fbaa00a98f0 2024-03-25T19:43:19.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.730+0000 7fba9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:19.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.730+0000 7fba9cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:19.730 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.730+0000 7fba9cff9640 10 monclient(hunting): dump: 2024-03-25T19:43:19.730 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:19.730 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:19.730 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:19.731 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:19.731 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:19.731 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:19.731 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:19.731 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:19.731 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:19.731 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:19.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.730+0000 7fba9cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:19.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.740+0000 7fba9cff9640 1 -- 172.21.15.44:0/1114698165 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2814551790 0 0) 0x7fbaa00abb00 con 0x7fbaa013f710 2024-03-25T19:43:19.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.740+0000 7fba9cff9640 1 -- 172.21.15.44:0/1114698165 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba8c004eb0 con 0x7fbaa013f710 2024-03-25T19:43:19.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.740+0000 7fba9cff9640 1 -- 172.21.15.44:0/1114698165 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fba94001b00 con 0x7fbaa013f710 2024-03-25T19:43:19.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.740+0000 7fba9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:19.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.740+0000 7fba9cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:19.741 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.740+0000 7fba9cff9640 10 monclient(hunting): dump: 2024-03-25T19:43:19.741 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:19.741 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:19.741 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:19.741 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:19.741 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:19.741 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:19.741 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:19.741 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:19.741 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:19.741 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:19.742 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.741+0000 7fba9cff9640 1 -- 172.21.15.44:0/1114698165 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 715307418 0 0) 0x7fbaa00b2a30 con 0x7fbaa00a98f0 2024-03-25T19:43:19.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.741+0000 7fba9cff9640 1 -- 172.21.15.44:0/1114698165 >> v1:172.21.15.111:6789/0 conn(0x7fbaa013f710 legacy=0x7fbaa00ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:19.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.741+0000 7fba9cff9640 1 -- 172.21.15.44:0/1114698165 >> v1:172.21.15.44:6789/0 conn(0x7fbaa01430c0 legacy=0x7fbaa01e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:19.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.741+0000 7fba9cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:43:19.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.741+0000 7fba9cff9640 1 monclient: found mon.c 2024-03-25T19:43:19.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.741+0000 7fba9cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:19.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.741+0000 7fba9cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:43:19.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.741+0000 7fba9cff9640 1 -- 172.21.15.44:0/1114698165 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa00abde0 con 0x7fbaa00a98f0 2024-03-25T19:43:19.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.741+0000 7fba9cff9640 10 monclient: _finish_auth 0 2024-03-25T19:43:19.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.741+0000 7fba9cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:43:19.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.741+0000 7fba9cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:19.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.741+0000 7fbaa68d9640 5 monclient: authenticate success, global_id 4568 2024-03-25T19:43:19.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.741+0000 7fbaa68d9640 10 monclient: _renew_subs 2024-03-25T19:43:19.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.741+0000 7fbaa68d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:43:19.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.741+0000 7fbaa68d9640 1 -- 172.21.15.44:0/1114698165 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa0138f40 con 0x7fbaa00a98f0 2024-03-25T19:43:19.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.741+0000 7fbaa68d9640 10 client.4568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:19.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.741+0000 7fbaa68d9640 10 monclient: _renew_subs 2024-03-25T19:43:19.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.741+0000 7fbaa68d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:43:19.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.741+0000 7fbaa68d9640 1 -- 172.21.15.44:0/1114698165 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa01363f0 con 0x7fbaa00a98f0 2024-03-25T19:43:19.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.741+0000 7fbaa68d9640 1 librados: init done 2024-03-25T19:43:19.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.743+0000 7fbaa68d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:19.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.743+0000 7fbaa68d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:19.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.743+0000 7fbaa68d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:43:19.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:19.743+0000 7fbaa68d9640 1 -- 172.21.15.44:0/1114698165 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa00627b0 con 0x7fbaa00a98f0 2024-03-25T19:43:20.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.062+0000 7fba9cff9640 1 -- 172.21.15.44:0/1114698165 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba900035f0 con 0x7fbaa00a98f0 2024-03-25T19:43:20.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.062+0000 7fba9cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:20.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.062+0000 7fba9cff9640 1 -- 172.21.15.44:0/1114698165 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fba90003980 con 0x7fbaa00a98f0 2024-03-25T19:43:20.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.062+0000 7fba9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:20.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.062+0000 7fba9cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:20.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.062+0000 7fba9cff9640 10 monclient: dump: 2024-03-25T19:43:20.063 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:20.064 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:20.064 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:20.064 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:20.064 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:20.064 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:20.064 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:20.064 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:20.064 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:20.064 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:20.064 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.062+0000 7fba9e7fc640 4 set_mon_vals no callback set 2024-03-25T19:43:20.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.063+0000 7fba9cff9640 1 -- 172.21.15.44:0/1114698165 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7fba90003ba0 con 0x7fbaa00a98f0 2024-03-25T19:43:20.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.064+0000 7fba9f7fe640 10 monclient: get_auth_request con 0x7fba8c06df70 auth_method 0 2024-03-25T19:43:20.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.064+0000 7fba9cff9640 10 client.4568.objecter ms_handle_connect 0x7fba8c06df70 2024-03-25T19:43:20.069 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.069+0000 7fba9cff9640 1 -- 172.21.15.44:0/1114698165 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7fba90053600 con 0x7fbaa00a98f0 2024-03-25T19:43:20.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.069+0000 7fba9cff9640 10 client.4568.objecter ms_dispatch 0x7fbaa0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:20.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.069+0000 7fba9cff9640 3 client.4568.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:20.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.069+0000 7fba9cff9640 3 client.4568.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:20.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.069+0000 7fba9cff9640 20 client.4568.objecter dump_active .. 0 homeless 2024-03-25T19:43:20.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.069+0000 7fba9cff9640 1 -- 172.21.15.44:0/1114698165 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fba9004f110 con 0x7fbaa00a98f0 2024-03-25T19:43:20.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.069+0000 7fba9cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:20.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.069+0000 7fba9cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:20.199 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:08:11.079 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:20.202 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:08:11.082 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:20.212 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:08:11.092 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:20.217 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:08:11.119 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:20.230 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:08:11.132 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:20.255 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:08:11.157 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:20.282 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:08:11.184 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:20.295 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:08:11.196 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:20.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.307+0000 7fbaa68d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:20.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.307+0000 7fbaa68d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:20.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.307+0000 7fbaa68d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:43:20.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.307+0000 7fbaa68d9640 1 -- 172.21.15.44:0/1114698165 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbaa01434a0 con 0x7fbaa00a98f0 2024-03-25T19:43:20.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.354+0000 7fba9cff9640 1 -- 172.21.15.44:0/1114698165 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7fba90051040 con 0x7fbaa00a98f0 2024-03-25T19:43:20.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.354+0000 7fba9cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:20.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.354+0000 7fba9cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:20.355 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:43:20.355 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:20.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.356+0000 7fbaa68d9640 10 librados: watch_flush enter 2024-03-25T19:43:20.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.356+0000 7fbaa68d9640 10 librados: watch_flush exit 2024-03-25T19:43:20.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.356+0000 7fbaa68d9640 20 client.4568.objecter shutdown clearing up homeless session... 2024-03-25T19:43:20.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.356+0000 7fbaa68d9640 10 client.4568.objecter successfully canceled tick 2024-03-25T19:43:20.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.356+0000 7fbaa68d9640 1 -- 172.21.15.44:0/1114698165 >> v1:172.21.15.111:6800/3883215793 conn(0x7fba8c06df70 legacy=0x7fba8c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:20.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.356+0000 7fbaa68d9640 10 monclient: shutdown 2024-03-25T19:43:20.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.356+0000 7fbaa68d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:20.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.356+0000 7fbaa68d9640 1 -- 172.21.15.44:0/1114698165 >> v1:172.21.15.44:6790/0 conn(0x7fbaa00a98f0 legacy=0x7fbaa0138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:20.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.356+0000 7fbaa68d9640 1 -- 172.21.15.44:0/1114698165 shutdown_connections 2024-03-25T19:43:20.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.356+0000 7fbaa68d9640 1 -- 172.21.15.44:0/1114698165 >> 172.21.15.44:0/1114698165 conn(0x7fbaa00b7ea0 msgr2=0x7fbaa0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:20.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.356+0000 7fbaa68d9640 1 -- 172.21.15.44:0/1114698165 shutdown_connections 2024-03-25T19:43:20.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.356+0000 7fbaa68d9640 1 -- 172.21.15.44:0/1114698165 wait complete. 2024-03-25T19:43:20.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:20.357+0000 7fbaa68d9640 1 librados: shutdown 2024-03-25T19:43:20.358 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:08:11.260 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:20.367 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:27.173 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:27.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.290+0000 7fcc4e081640 10 monclient: get_monmap_and_config 2024-03-25T19:43:27.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.290+0000 7fcc4e081640 10 monclient: build_initial_monmap 2024-03-25T19:43:27.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.290+0000 7fcc4e081640 10 monclient: monmap: 2024-03-25T19:43:27.291 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:43:27.291 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:27.291 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:43:27.291765+0000 2024-03-25T19:43:27.291 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:43:27.291765+0000 2024-03-25T19:43:27.291 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:27.291 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:27.291 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:43:27.291 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:43:27.291 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:43:27.291 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:27.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.291+0000 7fcc4e081640 1 Processor -- start 2024-03-25T19:43:27.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.291+0000 7fcc4e081640 1 -- start start 2024-03-25T19:43:27.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.291+0000 7fcc4e081640 10 monclient: init 2024-03-25T19:43:27.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.292+0000 7fcc4e081640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:27.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.292+0000 7fcc4e081640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:27.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.292+0000 7fcc4e081640 10 monclient(hunting): picked mon.noname-c con 0x7fcc480a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:43:27.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.292+0000 7fcc4e081640 10 monclient(hunting): picked mon.noname-b con 0x7fcc4813f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:43:27.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.292+0000 7fcc4e081640 10 monclient(hunting): picked mon.noname-a con 0x7fcc481430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:43:27.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.292+0000 7fcc4e081640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc48142520 con 0x7fcc481430c0 2024-03-25T19:43:27.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.292+0000 7fcc4e081640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc480b28a0 con 0x7fcc480a98f0 2024-03-25T19:43:27.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.292+0000 7fcc4e081640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc480b2a30 con 0x7fcc4813f710 2024-03-25T19:43:27.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.292+0000 7fcc4e081640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:27.294 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.292+0000 7fcc4e081640 10 monclient(hunting): authenticate will time out at 5871.463576s 2024-03-25T19:43:27.300 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.300+0000 7fcc477fe640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fcc480a98f0 0x7fcc480a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:45566/0 (socket says 172.21.15.44:45566) 2024-03-25T19:43:27.300 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.300+0000 7fcc477fe640 1 -- 172.21.15.44:0/2635284311 learned_addr learned my addr 172.21.15.44:0/2635284311 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:43:27.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.311+0000 7fcc46ffd640 1 -- 172.21.15.44:0/2635284311 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2144339736 0 0) 0x7fcc480b28a0 con 0x7fcc480a98f0 2024-03-25T19:43:27.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.311+0000 7fcc46ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:27.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.311+0000 7fcc46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:27.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.311+0000 7fcc46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:27.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.311+0000 7fcc46ffd640 10 monclient(hunting): my global_id is 4582 2024-03-25T19:43:27.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.311+0000 7fcc46ffd640 1 -- 172.21.15.44:0/2635284311 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc34003520 con 0x7fcc480a98f0 2024-03-25T19:43:27.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.320+0000 7fcc46ffd640 1 -- 172.21.15.44:0/2635284311 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3088541441 0 0) 0x7fcc48142520 con 0x7fcc481430c0 2024-03-25T19:43:27.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.320+0000 7fcc46ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:27.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.320+0000 7fcc46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:27.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.320+0000 7fcc46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:27.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.320+0000 7fcc46ffd640 10 monclient(hunting): my global_id is 4593 2024-03-25T19:43:27.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.320+0000 7fcc46ffd640 1 -- 172.21.15.44:0/2635284311 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc480b28a0 con 0x7fcc481430c0 2024-03-25T19:43:27.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.325+0000 7fcc46ffd640 1 -- 172.21.15.44:0/2635284311 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4097280778 0 0) 0x7fcc480b2a30 con 0x7fcc4813f710 2024-03-25T19:43:27.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.325+0000 7fcc46ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:27.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.325+0000 7fcc46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:27.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.325+0000 7fcc46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:27.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.325+0000 7fcc46ffd640 10 monclient(hunting): my global_id is 4571 2024-03-25T19:43:27.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.325+0000 7fcc46ffd640 1 -- 172.21.15.44:0/2635284311 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc48142520 con 0x7fcc4813f710 2024-03-25T19:43:27.361 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.361+0000 7fcc46ffd640 1 -- 172.21.15.44:0/2635284311 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1327360095 0 0) 0x7fcc480b28a0 con 0x7fcc481430c0 2024-03-25T19:43:27.361 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.361+0000 7fcc46ffd640 1 -- 172.21.15.44:0/2635284311 >> v1:172.21.15.44:6790/0 conn(0x7fcc4813f710 legacy=0x7fcc481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:27.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.361+0000 7fcc46ffd640 1 -- 172.21.15.44:0/2635284311 >> v1:172.21.15.111:6789/0 conn(0x7fcc480a98f0 legacy=0x7fcc480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:27.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.361+0000 7fcc46ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:27.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.361+0000 7fcc46ffd640 1 monclient: found mon.noname-a 2024-03-25T19:43:27.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.361+0000 7fcc46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:27.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.361+0000 7fcc46ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:43:27.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.361+0000 7fcc46ffd640 1 -- 172.21.15.44:0/2635284311 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc480b44c0 con 0x7fcc481430c0 2024-03-25T19:43:27.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.361+0000 7fcc46ffd640 10 monclient: _finish_auth 0 2024-03-25T19:43:27.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.361+0000 7fcc46ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:27.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.361+0000 7fcc46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:27.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.361+0000 7fcc46ffd640 1 -- 172.21.15.44:0/2635284311 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fcc400019f0 con 0x7fcc481430c0 2024-03-25T19:43:27.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.361+0000 7fcc4e081640 5 monclient: authenticate success, global_id 4593 2024-03-25T19:43:27.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.361+0000 7fcc4e081640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:27.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.361+0000 7fcc46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:27.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.361+0000 7fcc46ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:27.362 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.361+0000 7fcc46ffd640 10 monclient: dump: 2024-03-25T19:43:27.362 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:27.363 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:27.363 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:27.363 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:27.363 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:27.363 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:27.363 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:27.363 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:27.363 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:27.363 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:27.363 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.361+0000 7fcc4e081640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:27.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.431+0000 7fcc46ffd640 1 -- 172.21.15.44:0/2635284311 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc40001b50 con 0x7fcc481430c0 2024-03-25T19:43:27.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.431+0000 7fcc46ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:27.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.431+0000 7fcc46ffd640 1 -- 172.21.15.44:0/2635284311 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fcc40001f60 con 0x7fcc481430c0 2024-03-25T19:43:27.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.431+0000 7fcc46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:27.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.431+0000 7fcc46ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:27.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.431+0000 7fcc46ffd640 10 monclient: dump: 2024-03-25T19:43:27.432 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:27.432 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:27.432 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:27.432 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:27.432 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:27.432 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:27.432 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:27.432 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:27.432 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:27.432 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:27.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.431+0000 7fcc4e081640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:27.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.431+0000 7fcc4e081640 4 set_mon_vals no callback set 2024-03-25T19:43:27.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.432+0000 7fcc4e081640 10 monclient: shutdown 2024-03-25T19:43:27.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.432+0000 7fcc4e081640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:27.433 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.432+0000 7fcc4e081640 1 -- 172.21.15.44:0/2635284311 >> v1:172.21.15.44:6789/0 conn(0x7fcc481430c0 legacy=0x7fcc4814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:27.434 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.432+0000 7fcc4e081640 1 -- 172.21.15.44:0/2635284311 shutdown_connections 2024-03-25T19:43:27.434 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.432+0000 7fcc4e081640 1 -- 172.21.15.44:0/2635284311 >> 172.21.15.44:0/2635284311 conn(0x7fcc480b7ea0 msgr2=0x7fcc480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:27.434 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.432+0000 7fcc4e081640 1 -- 172.21.15.44:0/2635284311 shutdown_connections 2024-03-25T19:43:27.434 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.432+0000 7fcc4e081640 1 -- 172.21.15.44:0/2635284311 wait complete. 2024-03-25T19:43:27.434 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.433+0000 7fcc4e081640 10 monclient: build_initial_monmap 2024-03-25T19:43:27.434 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.433+0000 7fcc4e081640 10 monclient: monmap: 2024-03-25T19:43:27.434 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:43:27.434 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:27.434 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:43:27.434 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:43:27.434 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:27.434 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:43:27.434 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:43:27.434 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:43:27.434 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:43:27.435 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:27.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.433+0000 7fcc4e081640 1 librados: starting msgr at 2024-03-25T19:43:27.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.433+0000 7fcc4e081640 1 librados: starting objecter 2024-03-25T19:43:27.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.433+0000 7fcc4e081640 1 Processor -- start 2024-03-25T19:43:27.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.433+0000 7fcc4e081640 1 -- start start 2024-03-25T19:43:27.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.433+0000 7fcc4e081640 1 librados: setting wanted keys 2024-03-25T19:43:27.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.433+0000 7fcc4e081640 1 librados: calling monclient init 2024-03-25T19:43:27.435 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.433+0000 7fcc4e081640 10 monclient: init 2024-03-25T19:43:27.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.434+0000 7fcc4e081640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:27.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.434+0000 7fcc4e081640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:27.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.434+0000 7fcc4e081640 10 monclient(hunting): picked mon.noname-c con 0x7fcc480a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T19:43:27.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.434+0000 7fcc4e081640 10 monclient(hunting): picked mon.noname-a con 0x7fcc4813f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:43:27.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.434+0000 7fcc4e081640 10 monclient(hunting): picked mon.noname-b con 0x7fcc481430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:43:27.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.434+0000 7fcc4e081640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc48135bf0 con 0x7fcc4813f710 2024-03-25T19:43:27.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.434+0000 7fcc4e081640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc480b6d10 con 0x7fcc481430c0 2024-03-25T19:43:27.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.434+0000 7fcc4e081640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc481317f0 con 0x7fcc480a98f0 2024-03-25T19:43:27.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.434+0000 7fcc4e081640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:27.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.434+0000 7fcc4e081640 10 monclient(hunting): authenticate will time out at 5871.605357s 2024-03-25T19:43:27.454 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.454+0000 7fcc47fff640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fcc481430c0 0x7fcc481db370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:45626/0 (socket says 172.21.15.44:45626) 2024-03-25T19:43:27.454 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.454+0000 7fcc47fff640 1 -- 172.21.15.44:0/4049977321 learned_addr learned my addr 172.21.15.44:0/4049977321 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:43:27.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.458+0000 7fcc44ff9640 10 client.?.objecter ms_handle_connect 0x7fcc481430c0 2024-03-25T19:43:27.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.458+0000 7fcc44ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:27.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.465+0000 7fcc44ff9640 1 -- 172.21.15.44:0/4049977321 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3296778180 0 0) 0x7fcc480b6d10 con 0x7fcc481430c0 2024-03-25T19:43:27.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.465+0000 7fcc44ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:27.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.465+0000 7fcc44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:27.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.465+0000 7fcc44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:27.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.465+0000 7fcc44ff9640 10 monclient(hunting): my global_id is 4585 2024-03-25T19:43:27.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.465+0000 7fcc44ff9640 1 -- 172.21.15.44:0/4049977321 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc28004eb0 con 0x7fcc481430c0 2024-03-25T19:43:27.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.466+0000 7fcc44ff9640 10 client.?.objecter ms_handle_connect 0x7fcc4813f710 2024-03-25T19:43:27.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.466+0000 7fcc44ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:27.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.481+0000 7fcc44ff9640 1 -- 172.21.15.44:0/4049977321 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1458459445 0 0) 0x7fcc28004eb0 con 0x7fcc481430c0 2024-03-25T19:43:27.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.481+0000 7fcc44ff9640 1 -- 172.21.15.44:0/4049977321 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc480b6d10 con 0x7fcc481430c0 2024-03-25T19:43:27.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.481+0000 7fcc44ff9640 1 -- 172.21.15.44:0/4049977321 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fcc40001fa0 con 0x7fcc481430c0 2024-03-25T19:43:27.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.481+0000 7fcc44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:27.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.481+0000 7fcc44ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:27.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.481+0000 7fcc44ff9640 10 monclient(hunting): dump: 2024-03-25T19:43:27.481 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:27.482 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:27.482 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:27.482 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:27.482 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:27.482 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:27.482 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:27.482 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:27.482 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:27.482 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:27.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.481+0000 7fcc44ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:27.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.481+0000 7fcc44ff9640 1 -- 172.21.15.44:0/4049977321 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 413867021 0 0) 0x7fcc48135bf0 con 0x7fcc4813f710 2024-03-25T19:43:27.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.481+0000 7fcc44ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:27.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.481+0000 7fcc44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:27.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.481+0000 7fcc44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:27.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.481+0000 7fcc44ff9640 10 monclient(hunting): my global_id is 4596 2024-03-25T19:43:27.483 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.481+0000 7fcc44ff9640 1 -- 172.21.15.44:0/4049977321 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc28004eb0 con 0x7fcc4813f710 2024-03-25T19:43:27.483 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.482+0000 7fcc44ff9640 10 client.?.objecter ms_handle_connect 0x7fcc480a98f0 2024-03-25T19:43:27.483 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.482+0000 7fcc44ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:27.499 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.499+0000 7fcc44ff9640 1 -- 172.21.15.44:0/4049977321 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416245687 0 0) 0x7fcc481317f0 con 0x7fcc480a98f0 2024-03-25T19:43:27.499 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.499+0000 7fcc44ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:27.499 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.499+0000 7fcc44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:27.499 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.499+0000 7fcc44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:27.499 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.499+0000 7fcc44ff9640 10 monclient(hunting): my global_id is 4574 2024-03-25T19:43:27.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.499+0000 7fcc44ff9640 1 -- 172.21.15.44:0/4049977321 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc48135bf0 con 0x7fcc480a98f0 2024-03-25T19:43:27.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.505+0000 7fcc44ff9640 1 -- 172.21.15.44:0/4049977321 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2058348871 0 0) 0x7fcc480b6d10 con 0x7fcc481430c0 2024-03-25T19:43:27.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.505+0000 7fcc44ff9640 1 -- 172.21.15.44:0/4049977321 >> v1:172.21.15.44:6790/0 conn(0x7fcc480a98f0 legacy=0x7fcc481310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:27.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.505+0000 7fcc44ff9640 1 -- 172.21.15.44:0/4049977321 >> v1:172.21.15.44:6789/0 conn(0x7fcc4813f710 legacy=0x7fcc481dac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:27.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.505+0000 7fcc44ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:43:27.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.505+0000 7fcc44ff9640 1 monclient: found mon.b 2024-03-25T19:43:27.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.505+0000 7fcc44ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:27.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.505+0000 7fcc44ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:43:27.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.505+0000 7fcc44ff9640 1 -- 172.21.15.44:0/4049977321 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc48131980 con 0x7fcc481430c0 2024-03-25T19:43:27.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.505+0000 7fcc44ff9640 10 monclient: _finish_auth 0 2024-03-25T19:43:27.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.505+0000 7fcc44ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:43:27.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.505+0000 7fcc44ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:27.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.505+0000 7fcc4e081640 5 monclient: authenticate success, global_id 4585 2024-03-25T19:43:27.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.505+0000 7fcc4e081640 10 monclient: _renew_subs 2024-03-25T19:43:27.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.505+0000 7fcc4e081640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:43:27.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.505+0000 7fcc4e081640 1 -- 172.21.15.44:0/4049977321 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc480b76e0 con 0x7fcc481430c0 2024-03-25T19:43:27.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.505+0000 7fcc4e081640 10 client.4585.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:27.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.505+0000 7fcc4e081640 10 monclient: _renew_subs 2024-03-25T19:43:27.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.505+0000 7fcc4e081640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:43:27.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.505+0000 7fcc4e081640 1 -- 172.21.15.44:0/4049977321 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc48135270 con 0x7fcc481430c0 2024-03-25T19:43:27.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.505+0000 7fcc4e081640 1 librados: init done 2024-03-25T19:43:27.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.506+0000 7fcc4e081640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:27.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.506+0000 7fcc4e081640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:27.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.506+0000 7fcc4e081640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:43:27.508 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.506+0000 7fcc4e081640 1 -- 172.21.15.44:0/4049977321 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc480627b0 con 0x7fcc481430c0 2024-03-25T19:43:27.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.796+0000 7fcc44ff9640 1 -- 172.21.15.44:0/4049977321 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc40003c20 con 0x7fcc481430c0 2024-03-25T19:43:27.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.796+0000 7fcc44ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:27.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.796+0000 7fcc44ff9640 1 -- 172.21.15.44:0/4049977321 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fcc40003ff0 con 0x7fcc481430c0 2024-03-25T19:43:27.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.796+0000 7fcc44ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:27.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.796+0000 7fcc44ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:27.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.796+0000 7fcc44ff9640 10 monclient: dump: 2024-03-25T19:43:27.796 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:27.796 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:27.796 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:27.797 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:27.797 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:27.797 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:27.797 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:27.797 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:27.797 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:27.797 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:27.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.796+0000 7fcc467fc640 4 set_mon_vals no callback set 2024-03-25T19:43:27.798 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.796+0000 7fcc44ff9640 1 -- 172.21.15.44:0/4049977321 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7fcc400041d0 con 0x7fcc481430c0 2024-03-25T19:43:27.798 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.797+0000 7fcc477fe640 10 monclient: get_auth_request con 0x7fcc2806de10 auth_method 0 2024-03-25T19:43:27.798 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.797+0000 7fcc44ff9640 10 client.4585.objecter ms_handle_connect 0x7fcc2806de10 2024-03-25T19:43:27.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.801+0000 7fcc44ff9640 1 -- 172.21.15.44:0/4049977321 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7fcc40054c40 con 0x7fcc481430c0 2024-03-25T19:43:27.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.801+0000 7fcc44ff9640 10 client.4585.objecter ms_dispatch 0x7fcc48000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:27.802 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.801+0000 7fcc44ff9640 3 client.4585.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:27.802 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.801+0000 7fcc44ff9640 3 client.4585.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:27.802 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.801+0000 7fcc44ff9640 20 client.4585.objecter dump_active .. 0 homeless 2024-03-25T19:43:27.802 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.802+0000 7fcc44ff9640 1 -- 172.21.15.44:0/4049977321 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcc40050750 con 0x7fcc481430c0 2024-03-25T19:43:27.802 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.802+0000 7fcc44ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:27.802 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:27.802+0000 7fcc44ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:28.055 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:28.054+0000 7fcc4e081640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:28.055 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:28.054+0000 7fcc4e081640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:28.055 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:28.055+0000 7fcc4e081640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:43:28.055 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:28.055+0000 7fcc4e081640 1 -- 172.21.15.44:0/4049977321 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc4813e100 con 0x7fcc481430c0 2024-03-25T19:43:28.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:28.098+0000 7fcc44ff9640 1 -- 172.21.15.44:0/4049977321 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7fcc40052680 con 0x7fcc481430c0 2024-03-25T19:43:28.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:28.098+0000 7fcc44ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:28.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:28.098+0000 7fcc44ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:28.098 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:43:28.098 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:28.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:28.100+0000 7fcc4e081640 10 librados: watch_flush enter 2024-03-25T19:43:28.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:28.100+0000 7fcc4e081640 10 librados: watch_flush exit 2024-03-25T19:43:28.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:28.100+0000 7fcc4e081640 20 client.4585.objecter shutdown clearing up homeless session... 2024-03-25T19:43:28.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:28.100+0000 7fcc4e081640 10 client.4585.objecter successfully canceled tick 2024-03-25T19:43:28.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:28.100+0000 7fcc4e081640 1 -- 172.21.15.44:0/4049977321 >> v1:172.21.15.111:6800/3883215793 conn(0x7fcc2806de10 legacy=0x7fcc280702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:28.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:28.100+0000 7fcc4e081640 10 monclient: shutdown 2024-03-25T19:43:28.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:28.100+0000 7fcc4e081640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:28.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:28.100+0000 7fcc4e081640 1 -- 172.21.15.44:0/4049977321 >> v1:172.21.15.111:6789/0 conn(0x7fcc481430c0 legacy=0x7fcc481db370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:28.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:28.101+0000 7fcc4e081640 1 -- 172.21.15.44:0/4049977321 shutdown_connections 2024-03-25T19:43:28.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:28.101+0000 7fcc4e081640 1 -- 172.21.15.44:0/4049977321 >> 172.21.15.44:0/4049977321 conn(0x7fcc480b7ea0 msgr2=0x7fcc48074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:28.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:28.101+0000 7fcc4e081640 1 -- 172.21.15.44:0/4049977321 shutdown_connections 2024-03-25T19:43:28.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:28.101+0000 7fcc4e081640 1 -- 172.21.15.44:0/4049977321 wait complete. 2024-03-25T19:43:28.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:28.101+0000 7fcc4e081640 1 librados: shutdown 2024-03-25T19:43:28.112 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:34.919 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:35.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.033+0000 7fba3b8c0640 10 monclient: get_monmap_and_config 2024-03-25T19:43:35.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.033+0000 7fba3b8c0640 10 monclient: build_initial_monmap 2024-03-25T19:43:35.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.033+0000 7fba3b8c0640 10 monclient: monmap: 2024-03-25T19:43:35.032 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:43:35.032 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:35.032 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:43:35.033406+0000 2024-03-25T19:43:35.032 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:43:35.033406+0000 2024-03-25T19:43:35.032 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:35.032 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:35.032 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:43:35.033 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:43:35.033 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:43:35.033 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:35.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.033+0000 7fba3b8c0640 1 Processor -- start 2024-03-25T19:43:35.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.034+0000 7fba3b8c0640 1 -- start start 2024-03-25T19:43:35.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.034+0000 7fba3b8c0640 10 monclient: init 2024-03-25T19:43:35.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.034+0000 7fba3b8c0640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:35.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.034+0000 7fba3b8c0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:35.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.034+0000 7fba3b8c0640 10 monclient(hunting): picked mon.noname-a con 0x7fba340a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:43:35.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.034+0000 7fba3b8c0640 10 monclient(hunting): picked mon.noname-c con 0x7fba3413f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:43:35.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.034+0000 7fba3b8c0640 10 monclient(hunting): picked mon.noname-b con 0x7fba341430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:43:35.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.034+0000 7fba3b8c0640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba34142520 con 0x7fba340a98f0 2024-03-25T19:43:35.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.034+0000 7fba3b8c0640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba340b28a0 con 0x7fba3413f710 2024-03-25T19:43:35.035 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.034+0000 7fba3b8c0640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba340b2a30 con 0x7fba341430c0 2024-03-25T19:43:35.035 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.034+0000 7fba3b8c0640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:35.035 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.034+0000 7fba3b8c0640 10 monclient(hunting): authenticate will time out at 5879.204653s 2024-03-25T19:43:35.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.045+0000 7fba39e5d640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fba341430c0 0x7fba3414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:55596/0 (socket says 172.21.15.44:55596) 2024-03-25T19:43:35.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.045+0000 7fba39e5d640 1 -- 172.21.15.44:0/341558854 learned_addr learned my addr 172.21.15.44:0/341558854 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:43:35.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.063+0000 7fba2bfff640 1 -- 172.21.15.44:0/341558854 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468264562 0 0) 0x7fba34142520 con 0x7fba340a98f0 2024-03-25T19:43:35.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.063+0000 7fba2bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:35.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.063+0000 7fba2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:35.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.063+0000 7fba2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:35.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.063+0000 7fba2bfff640 10 monclient(hunting): my global_id is 4599 2024-03-25T19:43:35.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.063+0000 7fba2bfff640 1 -- 172.21.15.44:0/341558854 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba20003520 con 0x7fba340a98f0 2024-03-25T19:43:35.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.064+0000 7fba2bfff640 1 -- 172.21.15.44:0/341558854 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2219355186 0 0) 0x7fba340b2a30 con 0x7fba341430c0 2024-03-25T19:43:35.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.064+0000 7fba2bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:35.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.064+0000 7fba2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:35.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.064+0000 7fba2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:35.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.064+0000 7fba2bfff640 10 monclient(hunting): my global_id is 4577 2024-03-25T19:43:35.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.064+0000 7fba2bfff640 1 -- 172.21.15.44:0/341558854 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba34142520 con 0x7fba341430c0 2024-03-25T19:43:35.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.072+0000 7fba2bfff640 1 -- 172.21.15.44:0/341558854 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3254382049 0 0) 0x7fba340b28a0 con 0x7fba3413f710 2024-03-25T19:43:35.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.072+0000 7fba2bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:35.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.072+0000 7fba2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:35.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.072+0000 7fba2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:35.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.072+0000 7fba2bfff640 10 monclient(hunting): my global_id is 4588 2024-03-25T19:43:35.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.072+0000 7fba2bfff640 1 -- 172.21.15.44:0/341558854 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba340b2a30 con 0x7fba3413f710 2024-03-25T19:43:35.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.084+0000 7fba2bfff640 1 -- 172.21.15.44:0/341558854 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3664164492 0 0) 0x7fba340b2a30 con 0x7fba3413f710 2024-03-25T19:43:35.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.084+0000 7fba2bfff640 1 -- 172.21.15.44:0/341558854 >> v1:172.21.15.44:6790/0 conn(0x7fba341430c0 legacy=0x7fba3414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:35.083 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.084+0000 7fba2bfff640 1 -- 172.21.15.44:0/341558854 >> v1:172.21.15.44:6789/0 conn(0x7fba340a98f0 legacy=0x7fba340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:35.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.084+0000 7fba2bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:43:35.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.084+0000 7fba2bfff640 1 monclient: found mon.noname-c 2024-03-25T19:43:35.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.084+0000 7fba2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:35.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.084+0000 7fba2bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:43:35.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.084+0000 7fba2bfff640 1 -- 172.21.15.44:0/341558854 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba340b44c0 con 0x7fba3413f710 2024-03-25T19:43:35.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.084+0000 7fba2bfff640 10 monclient: _finish_auth 0 2024-03-25T19:43:35.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.084+0000 7fba2bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:43:35.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.084+0000 7fba2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:35.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.084+0000 7fba2bfff640 1 -- 172.21.15.44:0/341558854 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fba240015b0 con 0x7fba3413f710 2024-03-25T19:43:35.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.084+0000 7fba3b8c0640 5 monclient: authenticate success, global_id 4588 2024-03-25T19:43:35.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.084+0000 7fba3b8c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:35.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.084+0000 7fba2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:35.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.084+0000 7fba2bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:35.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.084+0000 7fba2bfff640 10 monclient: dump: 2024-03-25T19:43:35.084 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:35.085 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:35.085 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:35.085 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:35.085 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:35.085 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:35.085 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:35.085 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:35.085 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:35.085 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:35.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.084+0000 7fba3b8c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:35.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.097+0000 7fba2bfff640 1 -- 172.21.15.44:0/341558854 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba24001710 con 0x7fba3413f710 2024-03-25T19:43:35.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.097+0000 7fba2bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:35.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.097+0000 7fba2bfff640 1 -- 172.21.15.44:0/341558854 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fba24001b60 con 0x7fba3413f710 2024-03-25T19:43:35.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.097+0000 7fba2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:35.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.097+0000 7fba2bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:35.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.097+0000 7fba2bfff640 10 monclient: dump: 2024-03-25T19:43:35.097 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:35.097 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:35.097 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:35.097 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:35.097 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:35.097 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:35.097 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:35.097 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:35.098 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:35.098 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:35.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.097+0000 7fba3b8c0640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:35.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.097+0000 7fba3b8c0640 4 set_mon_vals no callback set 2024-03-25T19:43:35.100 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.098+0000 7fba3b8c0640 10 monclient: shutdown 2024-03-25T19:43:35.100 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.098+0000 7fba3b8c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:35.100 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.098+0000 7fba3b8c0640 1 -- 172.21.15.44:0/341558854 >> v1:172.21.15.111:6789/0 conn(0x7fba3413f710 legacy=0x7fba341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:35.100 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.098+0000 7fba3b8c0640 1 -- 172.21.15.44:0/341558854 shutdown_connections 2024-03-25T19:43:35.100 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.098+0000 7fba3b8c0640 1 -- 172.21.15.44:0/341558854 >> 172.21.15.44:0/341558854 conn(0x7fba340b7ea0 msgr2=0x7fba340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:35.100 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.098+0000 7fba3b8c0640 1 -- 172.21.15.44:0/341558854 shutdown_connections 2024-03-25T19:43:35.100 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.098+0000 7fba3b8c0640 1 -- 172.21.15.44:0/341558854 wait complete. 2024-03-25T19:43:35.100 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.099+0000 7fba3b8c0640 10 monclient: build_initial_monmap 2024-03-25T19:43:35.100 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.099+0000 7fba3b8c0640 10 monclient: monmap: 2024-03-25T19:43:35.100 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:43:35.100 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:35.100 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:43:35.100 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:43:35.101 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:35.101 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:43:35.101 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:43:35.101 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:43:35.101 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:43:35.101 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:35.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.099+0000 7fba3b8c0640 1 librados: starting msgr at 2024-03-25T19:43:35.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.099+0000 7fba3b8c0640 1 librados: starting objecter 2024-03-25T19:43:35.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.100+0000 7fba3b8c0640 1 Processor -- start 2024-03-25T19:43:35.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.100+0000 7fba3b8c0640 1 -- start start 2024-03-25T19:43:35.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.100+0000 7fba3b8c0640 1 librados: setting wanted keys 2024-03-25T19:43:35.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.100+0000 7fba3b8c0640 1 librados: calling monclient init 2024-03-25T19:43:35.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.100+0000 7fba3b8c0640 10 monclient: init 2024-03-25T19:43:35.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.100+0000 7fba3b8c0640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:35.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.100+0000 7fba3b8c0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:43:35.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.100+0000 7fba3b8c0640 10 monclient(hunting): picked mon.noname-a con 0x7fba340a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:43:35.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.100+0000 7fba3b8c0640 10 monclient(hunting): picked mon.noname-c con 0x7fba3413f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:43:35.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.100+0000 7fba3b8c0640 10 monclient(hunting): picked mon.noname-b con 0x7fba341430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:43:35.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.100+0000 7fba3b8c0640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba341db910 con 0x7fba340a98f0 2024-03-25T19:43:35.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.100+0000 7fba3b8c0640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba341db410 con 0x7fba341430c0 2024-03-25T19:43:35.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.100+0000 7fba3b8c0640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba341db5a0 con 0x7fba3413f710 2024-03-25T19:43:35.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.100+0000 7fba3b8c0640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:35.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.100+0000 7fba3b8c0640 10 monclient(hunting): authenticate will time out at 5879.270741s 2024-03-25T19:43:35.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.106+0000 7fba39e5d640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fba341430c0 0x7fba34137a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:59318/0 (socket says 172.21.15.44:59318) 2024-03-25T19:43:35.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.106+0000 7fba39e5d640 1 -- 172.21.15.44:0/744665306 learned_addr learned my addr 172.21.15.44:0/744665306 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:43:35.108 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.109+0000 7fba29ffb640 10 client.?.objecter ms_handle_connect 0x7fba341430c0 2024-03-25T19:43:35.108 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.109+0000 7fba29ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:35.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.115+0000 7fba29ffb640 1 -- 172.21.15.44:0/744665306 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2802349513 0 0) 0x7fba341db410 con 0x7fba341430c0 2024-03-25T19:43:35.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.115+0000 7fba29ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:35.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.115+0000 7fba29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:35.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.115+0000 7fba29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:35.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.115+0000 7fba29ffb640 10 monclient(hunting): my global_id is 4591 2024-03-25T19:43:35.114 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.115+0000 7fba29ffb640 1 -- 172.21.15.44:0/744665306 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba18004eb0 con 0x7fba341430c0 2024-03-25T19:43:35.121 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.122+0000 7fba29ffb640 10 client.?.objecter ms_handle_connect 0x7fba3413f710 2024-03-25T19:43:35.121 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.122+0000 7fba29ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:35.121 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.122+0000 7fba29ffb640 1 -- 172.21.15.44:0/744665306 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2257551192 0 0) 0x7fba18004eb0 con 0x7fba341430c0 2024-03-25T19:43:35.121 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.122+0000 7fba29ffb640 1 -- 172.21.15.44:0/744665306 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba341db410 con 0x7fba341430c0 2024-03-25T19:43:35.121 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.122+0000 7fba29ffb640 1 -- 172.21.15.44:0/744665306 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fba2c001c10 con 0x7fba341430c0 2024-03-25T19:43:35.121 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.122+0000 7fba29ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:35.121 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.122+0000 7fba29ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:35.121 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.122+0000 7fba29ffb640 10 monclient(hunting): dump: 2024-03-25T19:43:35.121 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:35.121 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:35.122 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:35.122 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:35.122 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:35.122 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:35.122 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:35.122 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:35.122 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:35.122 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:35.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.122+0000 7fba29ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:35.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.123+0000 7fba29ffb640 10 client.?.objecter ms_handle_connect 0x7fba340a98f0 2024-03-25T19:43:35.122 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.123+0000 7fba29ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:35.127 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.127+0000 7fba29ffb640 1 -- 172.21.15.44:0/744665306 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920753741 0 0) 0x7fba341db5a0 con 0x7fba3413f710 2024-03-25T19:43:35.127 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.127+0000 7fba29ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:35.127 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.127+0000 7fba29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:35.127 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.127+0000 7fba29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:35.127 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.127+0000 7fba29ffb640 10 monclient(hunting): my global_id is 4580 2024-03-25T19:43:35.127 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.127+0000 7fba29ffb640 1 -- 172.21.15.44:0/744665306 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba18004eb0 con 0x7fba3413f710 2024-03-25T19:43:35.127 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.127+0000 7fba29ffb640 1 -- 172.21.15.44:0/744665306 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1321554880 0 0) 0x7fba341db410 con 0x7fba341430c0 2024-03-25T19:43:35.127 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.127+0000 7fba29ffb640 1 -- 172.21.15.44:0/744665306 >> v1:172.21.15.44:6790/0 conn(0x7fba3413f710 legacy=0x7fba34133f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:35.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.127+0000 7fba29ffb640 1 -- 172.21.15.44:0/744665306 >> v1:172.21.15.44:6789/0 conn(0x7fba340a98f0 legacy=0x7fba341dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:35.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.127+0000 7fba29ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:43:35.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.127+0000 7fba29ffb640 1 monclient: found mon.b 2024-03-25T19:43:35.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.127+0000 7fba29ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:35.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.127+0000 7fba29ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:43:35.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.127+0000 7fba29ffb640 1 -- 172.21.15.44:0/744665306 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba34134690 con 0x7fba341430c0 2024-03-25T19:43:35.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.127+0000 7fba29ffb640 10 monclient: _finish_auth 0 2024-03-25T19:43:35.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.127+0000 7fba29ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:43:35.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.127+0000 7fba29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:35.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.128+0000 7fba3b8c0640 5 monclient: authenticate success, global_id 4591 2024-03-25T19:43:35.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.128+0000 7fba3b8c0640 10 monclient: _renew_subs 2024-03-25T19:43:35.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.128+0000 7fba3b8c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:43:35.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.128+0000 7fba3b8c0640 1 -- 172.21.15.44:0/744665306 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba34134960 con 0x7fba341430c0 2024-03-25T19:43:35.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.128+0000 7fba3b8c0640 10 client.4591.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:35.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.128+0000 7fba3b8c0640 10 monclient: _renew_subs 2024-03-25T19:43:35.128 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.128+0000 7fba3b8c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:43:35.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.128+0000 7fba3b8c0640 1 -- 172.21.15.44:0/744665306 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba34138b60 con 0x7fba341430c0 2024-03-25T19:43:35.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.128+0000 7fba3b8c0640 1 librados: init done 2024-03-25T19:43:35.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.129+0000 7fba3b8c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:35.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.129+0000 7fba3b8c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:35.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.129+0000 7fba3b8c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:43:35.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.129+0000 7fba3b8c0640 1 -- 172.21.15.44:0/744665306 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba34138eb0 con 0x7fba341430c0 2024-03-25T19:43:35.399 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.400+0000 7fba29ffb640 1 -- 172.21.15.44:0/744665306 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba2c004b60 con 0x7fba341430c0 2024-03-25T19:43:35.399 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.400+0000 7fba29ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:35.399 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.400+0000 7fba29ffb640 1 -- 172.21.15.44:0/744665306 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fba2c004e70 con 0x7fba341430c0 2024-03-25T19:43:35.400 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.400+0000 7fba29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:35.400 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.400+0000 7fba29ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:35.400 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.400+0000 7fba29ffb640 10 monclient: dump: 2024-03-25T19:43:35.400 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:35.400 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:35.400 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:35.400 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:35.400 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:35.400 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:35.400 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:35.400 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:35.400 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:35.400 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:35.400 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.400+0000 7fba2b7fe640 4 set_mon_vals no callback set 2024-03-25T19:43:35.401 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.401+0000 7fba29ffb640 1 -- 172.21.15.44:0/744665306 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7fba2c01ae40 con 0x7fba341430c0 2024-03-25T19:43:35.401 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.401+0000 7fba3965c640 10 monclient: get_auth_request con 0x7fba1806dd00 auth_method 0 2024-03-25T19:43:35.401 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.402+0000 7fba29ffb640 10 client.4591.objecter ms_handle_connect 0x7fba1806dd00 2024-03-25T19:43:35.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.406+0000 7fba29ffb640 1 -- 172.21.15.44:0/744665306 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7fba2c054b80 con 0x7fba341430c0 2024-03-25T19:43:35.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.406+0000 7fba29ffb640 10 client.4591.objecter ms_dispatch 0x7fba34000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:35.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.406+0000 7fba29ffb640 3 client.4591.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:35.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.406+0000 7fba29ffb640 3 client.4591.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:35.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.406+0000 7fba29ffb640 20 client.4591.objecter dump_active .. 0 homeless 2024-03-25T19:43:35.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.406+0000 7fba29ffb640 1 -- 172.21.15.44:0/744665306 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fba2c0504c0 con 0x7fba341430c0 2024-03-25T19:43:35.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.406+0000 7fba29ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:35.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.406+0000 7fba29ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:35.658 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.659+0000 7fba3b8c0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:35.659 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.659+0000 7fba3b8c0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:35.659 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.659+0000 7fba3b8c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:43:35.659 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.659+0000 7fba3b8c0640 1 -- 172.21.15.44:0/744665306 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba340627b0 con 0x7fba341430c0 2024-03-25T19:43:35.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.688+0000 7fba29ffb640 1 -- 172.21.15.44:0/744665306 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7fba2c0525c0 con 0x7fba341430c0 2024-03-25T19:43:35.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.688+0000 7fba29ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:35.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.688+0000 7fba29ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:35.688 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:43:35.688 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:35.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.690+0000 7fba3b8c0640 10 librados: watch_flush enter 2024-03-25T19:43:35.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.690+0000 7fba3b8c0640 10 librados: watch_flush exit 2024-03-25T19:43:35.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.690+0000 7fba3b8c0640 20 client.4591.objecter shutdown clearing up homeless session... 2024-03-25T19:43:35.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.690+0000 7fba3b8c0640 10 client.4591.objecter successfully canceled tick 2024-03-25T19:43:35.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.691+0000 7fba3b8c0640 1 -- 172.21.15.44:0/744665306 >> v1:172.21.15.111:6800/3883215793 conn(0x7fba1806dd00 legacy=0x7fba180701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:35.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.691+0000 7fba3b8c0640 10 monclient: shutdown 2024-03-25T19:43:35.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.691+0000 7fba3b8c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:35.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.691+0000 7fba3b8c0640 1 -- 172.21.15.44:0/744665306 >> v1:172.21.15.111:6789/0 conn(0x7fba341430c0 legacy=0x7fba34137a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:35.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.691+0000 7fba3b8c0640 1 -- 172.21.15.44:0/744665306 shutdown_connections 2024-03-25T19:43:35.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.691+0000 7fba3b8c0640 1 -- 172.21.15.44:0/744665306 >> 172.21.15.44:0/744665306 conn(0x7fba340b7ea0 msgr2=0x7fba34074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:35.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.691+0000 7fba3b8c0640 1 -- 172.21.15.44:0/744665306 shutdown_connections 2024-03-25T19:43:35.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.691+0000 7fba3b8c0640 1 -- 172.21.15.44:0/744665306 wait complete. 2024-03-25T19:43:35.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:35.691+0000 7fba3b8c0640 1 librados: shutdown 2024-03-25T19:43:35.701 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:40.179 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:08:31.081 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:40.230 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:08:31.132 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:40.260 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:08:31.161 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:40.280 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:08:31.160 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:40.287 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:08:31.167 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:40.289 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:08:31.191 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:40.298 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:08:31.200 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:40.324 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:08:31.203 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:40.332 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:08:31.233 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:42.506 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:42.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.609+0000 7f5adb98f640 10 monclient: get_monmap_and_config 2024-03-25T19:43:42.610 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.609+0000 7f5adb98f640 10 monclient: build_initial_monmap 2024-03-25T19:43:42.610 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.609+0000 7f5adb98f640 10 monclient: monmap: 2024-03-25T19:43:42.610 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:43:42.610 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:42.610 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:43:42.610797+0000 2024-03-25T19:43:42.610 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:43:42.610797+0000 2024-03-25T19:43:42.610 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:42.610 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:42.610 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:43:42.610 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:43:42.610 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:43:42.610 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:42.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.610+0000 7f5adb98f640 1 Processor -- start 2024-03-25T19:43:42.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.610+0000 7f5adb98f640 1 -- start start 2024-03-25T19:43:42.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.610+0000 7f5adb98f640 10 monclient: init 2024-03-25T19:43:42.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.611+0000 7f5adb98f640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:42.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.611+0000 7f5adb98f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:42.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.611+0000 7f5adb98f640 10 monclient(hunting): picked mon.noname-c con 0x7f5ad40b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T19:43:42.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.611+0000 7f5adb98f640 10 monclient(hunting): picked mon.noname-b con 0x7f5ad40b1730 addr v1:172.21.15.44:6790/0 2024-03-25T19:43:42.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.611+0000 7f5adb98f640 10 monclient(hunting): picked mon.noname-a con 0x7f5ad414be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:43:42.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.611+0000 7f5adb98f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad40af940 con 0x7f5ad414be90 2024-03-25T19:43:42.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.611+0000 7f5adb98f640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad40b0a70 con 0x7f5ad40b4d60 2024-03-25T19:43:42.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.611+0000 7f5adb98f640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad40b0d50 con 0x7f5ad40b1730 2024-03-25T19:43:42.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.611+0000 7f5adb98f640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:42.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.611+0000 7f5adb98f640 10 monclient(hunting): authenticate will time out at 5886.782305s 2024-03-25T19:43:42.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.616+0000 7f5ad9f2c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f5ad414be90 0x7f5ad414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57474/0 (socket says 172.21.15.44:57474) 2024-03-25T19:43:42.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.616+0000 7f5ad9f2c640 1 -- 172.21.15.44:0/2121338011 learned_addr learned my addr 172.21.15.44:0/2121338011 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:43:42.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.637+0000 7f5acbfff640 1 -- 172.21.15.44:0/2121338011 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3619743631 0 0) 0x7f5ad40af940 con 0x7f5ad414be90 2024-03-25T19:43:42.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.637+0000 7f5acbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:42.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.637+0000 7f5acbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:42.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.637+0000 7f5acbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:42.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.637+0000 7f5acbfff640 10 monclient(hunting): my global_id is 4605 2024-03-25T19:43:42.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.637+0000 7f5acbfff640 1 -- 172.21.15.44:0/2121338011 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ac0003520 con 0x7f5ad414be90 2024-03-25T19:43:42.644 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.644+0000 7f5acbfff640 1 -- 172.21.15.44:0/2121338011 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2772158791 0 0) 0x7f5ad40b0d50 con 0x7f5ad40b1730 2024-03-25T19:43:42.644 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.644+0000 7f5acbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:42.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.644+0000 7f5acbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:42.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.644+0000 7f5acbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:42.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.644+0000 7f5acbfff640 10 monclient(hunting): my global_id is 4583 2024-03-25T19:43:42.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.644+0000 7f5acbfff640 1 -- 172.21.15.44:0/2121338011 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ad40af940 con 0x7f5ad40b1730 2024-03-25T19:43:42.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.646+0000 7f5acbfff640 1 -- 172.21.15.44:0/2121338011 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2897326077 0 0) 0x7f5ac0003520 con 0x7f5ad414be90 2024-03-25T19:43:42.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.646+0000 7f5acbfff640 1 -- 172.21.15.44:0/2121338011 >> v1:172.21.15.44:6790/0 conn(0x7f5ad40b1730 legacy=0x7f5ad40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:42.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.646+0000 7f5acbfff640 1 -- 172.21.15.44:0/2121338011 >> v1:172.21.15.111:6789/0 conn(0x7f5ad40b4d60 legacy=0x7f5ad413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:42.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.646+0000 7f5acbfff640 10 monclient: _finish_hunting 0 2024-03-25T19:43:42.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.646+0000 7f5acbfff640 1 monclient: found mon.noname-a 2024-03-25T19:43:42.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.646+0000 7f5acbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:42.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.646+0000 7f5acbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:43:42.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.646+0000 7f5acbfff640 1 -- 172.21.15.44:0/2121338011 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad40b1030 con 0x7f5ad414be90 2024-03-25T19:43:42.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.646+0000 7f5acbfff640 10 monclient: _finish_auth 0 2024-03-25T19:43:42.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.646+0000 7f5acbfff640 10 monclient: _check_auth_tickets 2024-03-25T19:43:42.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.646+0000 7f5acbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:42.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.646+0000 7f5acbfff640 1 -- 172.21.15.44:0/2121338011 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f5acc001a10 con 0x7f5ad414be90 2024-03-25T19:43:42.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.646+0000 7f5adb98f640 5 monclient: authenticate success, global_id 4605 2024-03-25T19:43:42.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.646+0000 7f5adb98f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:42.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.647+0000 7f5acbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:42.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.647+0000 7f5acbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:42.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.647+0000 7f5acbfff640 10 monclient: dump: 2024-03-25T19:43:42.648 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:42.648 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:42.648 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:42.648 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:42.648 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:42.648 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:42.649 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:42.649 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:42.649 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:42.649 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:42.649 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.647+0000 7f5adb98f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:42.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.676+0000 7f5acbfff640 1 -- 172.21.15.44:0/2121338011 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5acc001b70 con 0x7f5ad414be90 2024-03-25T19:43:42.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.676+0000 7f5acbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:42.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.676+0000 7f5acbfff640 1 -- 172.21.15.44:0/2121338011 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f5acc001f80 con 0x7f5ad414be90 2024-03-25T19:43:42.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.676+0000 7f5adb98f640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:42.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.676+0000 7f5adb98f640 4 set_mon_vals no callback set 2024-03-25T19:43:42.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.676+0000 7f5acbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:42.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.676+0000 7f5acbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:42.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.676+0000 7f5acbfff640 10 monclient: dump: 2024-03-25T19:43:42.677 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:42.677 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:42.677 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:42.677 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:42.677 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:42.677 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:42.677 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:42.677 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:42.677 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:42.677 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:42.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.677+0000 7f5adb98f640 10 monclient: shutdown 2024-03-25T19:43:42.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.677+0000 7f5adb98f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:42.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.677+0000 7f5adb98f640 1 -- 172.21.15.44:0/2121338011 >> v1:172.21.15.44:6789/0 conn(0x7f5ad414be90 legacy=0x7f5ad414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:42.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.677+0000 7f5adb98f640 1 -- 172.21.15.44:0/2121338011 shutdown_connections 2024-03-25T19:43:42.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.677+0000 7f5adb98f640 1 -- 172.21.15.44:0/2121338011 >> 172.21.15.44:0/2121338011 conn(0x7f5ad40617a0 msgr2=0x7f5ad4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:42.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.677+0000 7f5adb98f640 1 -- 172.21.15.44:0/2121338011 shutdown_connections 2024-03-25T19:43:42.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.677+0000 7f5adb98f640 1 -- 172.21.15.44:0/2121338011 wait complete. 2024-03-25T19:43:42.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.678+0000 7f5adb98f640 10 monclient: build_initial_monmap 2024-03-25T19:43:42.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.678+0000 7f5adb98f640 10 monclient: monmap: 2024-03-25T19:43:42.679 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:43:42.679 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:42.680 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:43:42.680 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:43:42.680 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:42.680 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:43:42.680 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:43:42.680 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:43:42.680 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:43:42.680 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:42.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.678+0000 7f5adb98f640 1 librados: starting msgr at 2024-03-25T19:43:42.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.678+0000 7f5adb98f640 1 librados: starting objecter 2024-03-25T19:43:42.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.679+0000 7f5adb98f640 1 Processor -- start 2024-03-25T19:43:42.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.679+0000 7f5adb98f640 1 -- start start 2024-03-25T19:43:42.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.679+0000 7f5adb98f640 1 librados: setting wanted keys 2024-03-25T19:43:42.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.679+0000 7f5adb98f640 1 librados: calling monclient init 2024-03-25T19:43:42.680 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.679+0000 7f5adb98f640 10 monclient: init 2024-03-25T19:43:42.682 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.679+0000 7f5adb98f640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:42.682 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.679+0000 7f5adb98f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:42.682 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.680+0000 7f5adb98f640 10 monclient(hunting): picked mon.noname-b con 0x7f5ad40b1730 addr v1:172.21.15.111:6789/0 2024-03-25T19:43:42.682 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.680+0000 7f5adb98f640 10 monclient(hunting): picked mon.noname-c con 0x7f5ad40b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T19:43:42.682 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.680+0000 7f5adb98f640 10 monclient(hunting): picked mon.noname-a con 0x7f5ad414be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:43:42.682 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.680+0000 7f5adb98f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad4142710 con 0x7f5ad414be90 2024-03-25T19:43:42.682 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.680+0000 7f5adb98f640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad4142410 con 0x7f5ad40b1730 2024-03-25T19:43:42.682 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.680+0000 7f5adb98f640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad4140d00 con 0x7f5ad40b4d60 2024-03-25T19:43:42.682 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.680+0000 7f5adb98f640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:42.682 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.680+0000 7f5adb98f640 10 monclient(hunting): authenticate will time out at 5886.851110s 2024-03-25T19:43:42.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.692+0000 7f5ad8f2a640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f5ad40b4d60 0x7f5ad4142ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:47788/0 (socket says 172.21.15.44:47788) 2024-03-25T19:43:42.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.692+0000 7f5ad8f2a640 1 -- 172.21.15.44:0/41682611 learned_addr learned my addr 172.21.15.44:0/41682611 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:43:42.696 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.696+0000 7f5ac9ffb640 10 client.?.objecter ms_handle_connect 0x7f5ad414be90 2024-03-25T19:43:42.696 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.696+0000 7f5ac9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:42.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.698+0000 7f5ac9ffb640 10 client.?.objecter ms_handle_connect 0x7f5ad40b4d60 2024-03-25T19:43:42.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.698+0000 7f5ac9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:42.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.700+0000 7f5ac9ffb640 10 client.?.objecter ms_handle_connect 0x7f5ad40b1730 2024-03-25T19:43:42.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.700+0000 7f5ac9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:42.708 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.708+0000 7f5ac9ffb640 1 -- 172.21.15.44:0/41682611 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69878992 0 0) 0x7f5ad4140d00 con 0x7f5ad40b4d60 2024-03-25T19:43:42.708 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.708+0000 7f5ac9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:42.708 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.708+0000 7f5ac9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:42.708 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.708+0000 7f5ac9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:42.708 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.708+0000 7f5ac9ffb640 10 monclient(hunting): my global_id is 4586 2024-03-25T19:43:42.708 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.708+0000 7f5ac9ffb640 1 -- 172.21.15.44:0/41682611 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab8004eb0 con 0x7f5ad40b4d60 2024-03-25T19:43:42.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.712+0000 7f5ac9ffb640 1 -- 172.21.15.44:0/41682611 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3427849635 0 0) 0x7f5ad4142710 con 0x7f5ad414be90 2024-03-25T19:43:42.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.712+0000 7f5ac9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:42.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.712+0000 7f5ac9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:42.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.712+0000 7f5ac9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:42.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.712+0000 7f5ac9ffb640 10 monclient(hunting): my global_id is 4608 2024-03-25T19:43:42.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.712+0000 7f5ac9ffb640 1 -- 172.21.15.44:0/41682611 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ad4140d00 con 0x7f5ad414be90 2024-03-25T19:43:42.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.719+0000 7f5ac9ffb640 1 -- 172.21.15.44:0/41682611 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3701761911 0 0) 0x7f5ab8004eb0 con 0x7f5ad40b4d60 2024-03-25T19:43:42.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.719+0000 7f5ac9ffb640 1 -- 172.21.15.44:0/41682611 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ad4142710 con 0x7f5ad40b4d60 2024-03-25T19:43:42.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.719+0000 7f5ac9ffb640 1 -- 172.21.15.44:0/41682611 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f5ac4001a10 con 0x7f5ad40b4d60 2024-03-25T19:43:42.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.719+0000 7f5ac9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:42.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.719+0000 7f5ac9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:42.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.719+0000 7f5ac9ffb640 10 monclient(hunting): dump: 2024-03-25T19:43:42.720 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:42.720 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:42.720 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:42.720 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:42.720 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:42.720 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:42.721 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:42.721 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:42.721 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:42.721 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:42.721 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.720+0000 7f5ac9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:42.721 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.720+0000 7f5ac9ffb640 1 -- 172.21.15.44:0/41682611 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824259616 0 0) 0x7f5ad4142410 con 0x7f5ad40b1730 2024-03-25T19:43:42.721 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.720+0000 7f5ac9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:42.721 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.720+0000 7f5ac9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:42.722 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.720+0000 7f5ac9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:42.722 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.720+0000 7f5ac9ffb640 10 monclient(hunting): my global_id is 4597 2024-03-25T19:43:42.722 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.720+0000 7f5ac9ffb640 1 -- 172.21.15.44:0/41682611 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab8004eb0 con 0x7f5ad40b1730 2024-03-25T19:43:42.722 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.722+0000 7f5ac9ffb640 1 -- 172.21.15.44:0/41682611 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3464778343 0 0) 0x7f5ad4140d00 con 0x7f5ad414be90 2024-03-25T19:43:42.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.722+0000 7f5ac9ffb640 1 -- 172.21.15.44:0/41682611 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ad4142410 con 0x7f5ad414be90 2024-03-25T19:43:42.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.722+0000 7f5ac9ffb640 1 -- 172.21.15.44:0/41682611 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f5acc001e80 con 0x7f5ad414be90 2024-03-25T19:43:42.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.722+0000 7f5ac9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:42.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.722+0000 7f5ac9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:42.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.722+0000 7f5ac9ffb640 10 monclient(hunting): dump: 2024-03-25T19:43:42.723 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:42.723 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:42.723 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:42.723 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:42.723 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:42.723 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:42.723 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:42.723 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:42.723 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:42.723 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:42.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.734+0000 7f5ac9ffb640 1 -- 172.21.15.44:0/41682611 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2918341482 0 0) 0x7f5ad4142410 con 0x7f5ad414be90 2024-03-25T19:43:42.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.734+0000 7f5ac9ffb640 1 -- 172.21.15.44:0/41682611 >> v1:172.21.15.44:6790/0 conn(0x7f5ad40b4d60 legacy=0x7f5ad4142ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:42.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.734+0000 7f5ac9ffb640 1 -- 172.21.15.44:0/41682611 >> v1:172.21.15.111:6789/0 conn(0x7f5ad40b1730 legacy=0x7f5ad41405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:42.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.734+0000 7f5ac9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:43:42.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.734+0000 7f5ac9ffb640 1 monclient: found mon.a 2024-03-25T19:43:42.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.734+0000 7f5ac9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:42.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.734+0000 7f5ac9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:43:42.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.734+0000 7f5ac9ffb640 1 -- 172.21.15.44:0/41682611 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad4139000 con 0x7f5ad414be90 2024-03-25T19:43:42.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.734+0000 7f5ac9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:43:42.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.734+0000 7f5ac9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:43:42.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.734+0000 7f5ac9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:42.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.734+0000 7f5adb98f640 5 monclient: authenticate success, global_id 4608 2024-03-25T19:43:42.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.734+0000 7f5adb98f640 10 monclient: _renew_subs 2024-03-25T19:43:42.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.734+0000 7f5adb98f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:43:42.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.734+0000 7f5adb98f640 1 -- 172.21.15.44:0/41682611 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad4141c30 con 0x7f5ad414be90 2024-03-25T19:43:42.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.734+0000 7f5adb98f640 10 client.4608.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:42.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.734+0000 7f5adb98f640 10 monclient: _renew_subs 2024-03-25T19:43:42.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.734+0000 7f5adb98f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:43:42.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.734+0000 7f5adb98f640 1 -- 172.21.15.44:0/41682611 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ad413b9d0 con 0x7f5ad414be90 2024-03-25T19:43:42.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.734+0000 7f5adb98f640 1 librados: init done 2024-03-25T19:43:42.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.735+0000 7f5adb98f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:42.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.736+0000 7f5adb98f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:42.736 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.736+0000 7f5adb98f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:43:42.737 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:42.736+0000 7f5adb98f640 1 -- 172.21.15.44:0/41682611 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad41428c0 con 0x7f5ad414be90 2024-03-25T19:43:43.048 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.048+0000 7f5ac9ffb640 1 -- 172.21.15.44:0/41682611 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5acc0015c0 con 0x7f5ad414be90 2024-03-25T19:43:43.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.048+0000 7f5ac9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:43.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.048+0000 7f5ac9ffb640 1 -- 172.21.15.44:0/41682611 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f5acc004e10 con 0x7f5ad414be90 2024-03-25T19:43:43.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.048+0000 7f5ac9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:43.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.048+0000 7f5ac9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:43.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.048+0000 7f5ac9ffb640 10 monclient: dump: 2024-03-25T19:43:43.049 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:43.049 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:43.049 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:43.049 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:43.049 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:43.049 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:43.050 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:43.050 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:43.050 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:43.050 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:43.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.048+0000 7f5acb7fe640 4 set_mon_vals no callback set 2024-03-25T19:43:43.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.049+0000 7f5ac9ffb640 1 -- 172.21.15.44:0/41682611 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f5acc01ade0 con 0x7f5ad414be90 2024-03-25T19:43:43.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.050+0000 7f5ad972b640 10 monclient: get_auth_request con 0x7f5ab806df00 auth_method 0 2024-03-25T19:43:43.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.050+0000 7f5ac9ffb640 10 client.4608.objecter ms_handle_connect 0x7f5ab806df00 2024-03-25T19:43:43.056 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.056+0000 7f5ac9ffb640 1 -- 172.21.15.44:0/41682611 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f5acc0549f0 con 0x7f5ad414be90 2024-03-25T19:43:43.056 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.056+0000 7f5ac9ffb640 10 client.4608.objecter ms_dispatch 0x7f5ad4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:43.056 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.056+0000 7f5ac9ffb640 3 client.4608.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:43.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.056+0000 7f5ac9ffb640 3 client.4608.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:43.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.056+0000 7f5ac9ffb640 20 client.4608.objecter dump_active .. 0 homeless 2024-03-25T19:43:43.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.056+0000 7f5ac9ffb640 1 -- 172.21.15.44:0/41682611 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5acc050330 con 0x7f5ad414be90 2024-03-25T19:43:43.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.056+0000 7f5ac9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:43.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.056+0000 7f5ac9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:43.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.312+0000 7f5adb98f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:43.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.312+0000 7f5adb98f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:43.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.312+0000 7f5adb98f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:43:43.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.312+0000 7f5adb98f640 1 -- 172.21.15.44:0/41682611 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ad40627b0 con 0x7f5ad414be90 2024-03-25T19:43:43.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.340+0000 7f5ac9ffb640 1 -- 172.21.15.44:0/41682611 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f5acc052430 con 0x7f5ad414be90 2024-03-25T19:43:43.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.340+0000 7f5ac9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:43.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.340+0000 7f5ac9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:43.341 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:43:43.341 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:43.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.342+0000 7f5adb98f640 10 librados: watch_flush enter 2024-03-25T19:43:43.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.342+0000 7f5adb98f640 10 librados: watch_flush exit 2024-03-25T19:43:43.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.342+0000 7f5adb98f640 20 client.4608.objecter shutdown clearing up homeless session... 2024-03-25T19:43:43.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.342+0000 7f5adb98f640 10 client.4608.objecter successfully canceled tick 2024-03-25T19:43:43.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.343+0000 7f5adb98f640 1 -- 172.21.15.44:0/41682611 >> v1:172.21.15.111:6800/3883215793 conn(0x7f5ab806df00 legacy=0x7f5ab80703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:43.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.343+0000 7f5adb98f640 10 monclient: shutdown 2024-03-25T19:43:43.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.343+0000 7f5adb98f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:43.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.343+0000 7f5adb98f640 1 -- 172.21.15.44:0/41682611 >> v1:172.21.15.44:6789/0 conn(0x7f5ad414be90 legacy=0x7f5ad41e2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:43.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.343+0000 7f5adb98f640 1 -- 172.21.15.44:0/41682611 shutdown_connections 2024-03-25T19:43:43.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.343+0000 7f5adb98f640 1 -- 172.21.15.44:0/41682611 >> 172.21.15.44:0/41682611 conn(0x7f5ad40617a0 msgr2=0x7f5ad40b6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:43.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.343+0000 7f5adb98f640 1 -- 172.21.15.44:0/41682611 shutdown_connections 2024-03-25T19:43:43.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.343+0000 7f5adb98f640 1 -- 172.21.15.44:0/41682611 wait complete. 2024-03-25T19:43:43.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:43.343+0000 7f5adb98f640 1 librados: shutdown 2024-03-25T19:43:43.355 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:50.162 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:50.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.276+0000 7fdd00f9e640 10 monclient: get_monmap_and_config 2024-03-25T19:43:50.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.276+0000 7fdd00f9e640 10 monclient: build_initial_monmap 2024-03-25T19:43:50.276 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.276+0000 7fdd00f9e640 10 monclient: monmap: 2024-03-25T19:43:50.276 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:43:50.277 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:50.277 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:43:50.277650+0000 2024-03-25T19:43:50.277 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:43:50.277650+0000 2024-03-25T19:43:50.277 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:50.277 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:50.277 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:43:50.277 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:43:50.277 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:43:50.277 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:50.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.277+0000 7fdd00f9e640 1 Processor -- start 2024-03-25T19:43:50.278 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.277+0000 7fdd00f9e640 1 -- start start 2024-03-25T19:43:50.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.277+0000 7fdd00f9e640 10 monclient: init 2024-03-25T19:43:50.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.278+0000 7fdd00f9e640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:50.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.278+0000 7fdd00f9e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:50.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.278+0000 7fdd00f9e640 10 monclient(hunting): picked mon.noname-a con 0x7fdcfc0b49a0 addr v1:172.21.15.44:6789/0 2024-03-25T19:43:50.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.278+0000 7fdd00f9e640 10 monclient(hunting): picked mon.noname-c con 0x7fdcfc0a9680 addr v1:172.21.15.111:6789/0 2024-03-25T19:43:50.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.278+0000 7fdd00f9e640 10 monclient(hunting): picked mon.noname-b con 0x7fdcfc0b3fa0 addr v1:172.21.15.44:6790/0 2024-03-25T19:43:50.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.278+0000 7fdd00f9e640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcfc13fe70 con 0x7fdcfc0b49a0 2024-03-25T19:43:50.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.278+0000 7fdd00f9e640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcfc0ab250 con 0x7fdcfc0a9680 2024-03-25T19:43:50.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.278+0000 7fdd00f9e640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcfc0a9b10 con 0x7fdcfc0b3fa0 2024-03-25T19:43:50.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.278+0000 7fdd00f9e640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:50.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.278+0000 7fdd00f9e640 10 monclient(hunting): authenticate will time out at 5894.449160s 2024-03-25T19:43:50.283 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.283+0000 7fdcfaffd640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fdcfc0a9680 0x7fdcfc145850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:51790/0 (socket says 172.21.15.44:51790) 2024-03-25T19:43:50.283 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.283+0000 7fdcfaffd640 1 -- 172.21.15.44:0/4123914016 learned_addr learned my addr 172.21.15.44:0/4123914016 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:43:50.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.291+0000 7fdcfa7fc640 1 -- 172.21.15.44:0/4123914016 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252400505 0 0) 0x7fdcfc0ab250 con 0x7fdcfc0a9680 2024-03-25T19:43:50.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.291+0000 7fdcfa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:50.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.291+0000 7fdcfa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:50.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.291+0000 7fdcfa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:50.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.291+0000 7fdcfa7fc640 10 monclient(hunting): my global_id is 4600 2024-03-25T19:43:50.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.291+0000 7fdcfa7fc640 1 -- 172.21.15.44:0/4123914016 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdce4003520 con 0x7fdcfc0a9680 2024-03-25T19:43:50.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.297+0000 7fdcfa7fc640 1 -- 172.21.15.44:0/4123914016 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1974517529 0 0) 0x7fdce4003520 con 0x7fdcfc0a9680 2024-03-25T19:43:50.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.297+0000 7fdcfa7fc640 1 -- 172.21.15.44:0/4123914016 >> v1:172.21.15.44:6790/0 conn(0x7fdcfc0b3fa0 legacy=0x7fdcfc145f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:50.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.297+0000 7fdcfa7fc640 1 -- 172.21.15.44:0/4123914016 >> v1:172.21.15.44:6789/0 conn(0x7fdcfc0b49a0 legacy=0x7fdcfc13f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:50.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.297+0000 7fdcfa7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:43:50.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.297+0000 7fdcfa7fc640 1 monclient: found mon.noname-c 2024-03-25T19:43:50.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.297+0000 7fdcfa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:50.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.297+0000 7fdcfa7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:43:50.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.297+0000 7fdcfa7fc640 1 -- 172.21.15.44:0/4123914016 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcfc151b50 con 0x7fdcfc0a9680 2024-03-25T19:43:50.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.297+0000 7fdcfa7fc640 10 monclient: _finish_auth 0 2024-03-25T19:43:50.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.297+0000 7fdcfa7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:43:50.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.297+0000 7fdcfa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:50.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.297+0000 7fdcfa7fc640 1 -- 172.21.15.44:0/4123914016 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fdcf0001690 con 0x7fdcfc0a9680 2024-03-25T19:43:50.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.297+0000 7fdd00f9e640 5 monclient: authenticate success, global_id 4600 2024-03-25T19:43:50.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.297+0000 7fdd00f9e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:50.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.297+0000 7fdcfa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:50.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.297+0000 7fdcfa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:50.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.297+0000 7fdcfa7fc640 10 monclient: dump: 2024-03-25T19:43:50.298 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:50.298 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:50.298 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:50.298 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:50.298 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:50.298 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:50.299 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:50.299 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:50.299 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:50.299 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:50.299 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.297+0000 7fdd00f9e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:50.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.301+0000 7fdcfa7fc640 1 -- 172.21.15.44:0/4123914016 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcf00017f0 con 0x7fdcfc0a9680 2024-03-25T19:43:50.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.301+0000 7fdcfa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:50.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.301+0000 7fdcfa7fc640 1 -- 172.21.15.44:0/4123914016 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fdcf0001b80 con 0x7fdcfc0a9680 2024-03-25T19:43:50.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.301+0000 7fdd00f9e640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:50.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.301+0000 7fdd00f9e640 4 set_mon_vals no callback set 2024-03-25T19:43:50.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.301+0000 7fdcfa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:50.301 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.301+0000 7fdcfa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:50.302 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.301+0000 7fdcfa7fc640 10 monclient: dump: 2024-03-25T19:43:50.302 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:50.302 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:50.302 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:50.302 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:50.302 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:50.302 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:50.302 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:50.302 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:50.302 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:50.302 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:50.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.302+0000 7fdd00f9e640 10 monclient: shutdown 2024-03-25T19:43:50.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.302+0000 7fdd00f9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:50.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.302+0000 7fdd00f9e640 1 -- 172.21.15.44:0/4123914016 >> v1:172.21.15.111:6789/0 conn(0x7fdcfc0a9680 legacy=0x7fdcfc145850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:50.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.302+0000 7fdd00f9e640 1 -- 172.21.15.44:0/4123914016 shutdown_connections 2024-03-25T19:43:50.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.302+0000 7fdd00f9e640 1 -- 172.21.15.44:0/4123914016 >> 172.21.15.44:0/4123914016 conn(0x7fdcfc0615c0 msgr2=0x7fdcfc0619a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:50.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.302+0000 7fdd00f9e640 1 -- 172.21.15.44:0/4123914016 shutdown_connections 2024-03-25T19:43:50.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.302+0000 7fdd00f9e640 1 -- 172.21.15.44:0/4123914016 wait complete. 2024-03-25T19:43:50.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.303+0000 7fdd00f9e640 10 monclient: build_initial_monmap 2024-03-25T19:43:50.304 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.303+0000 7fdd00f9e640 10 monclient: monmap: 2024-03-25T19:43:50.304 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:43:50.304 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:50.304 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:43:50.304 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:43:50.305 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:50.305 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:43:50.305 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:43:50.305 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:43:50.305 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:43:50.305 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:50.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.303+0000 7fdd00f9e640 1 librados: starting msgr at 2024-03-25T19:43:50.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.303+0000 7fdd00f9e640 1 librados: starting objecter 2024-03-25T19:43:50.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.303+0000 7fdd00f9e640 1 Processor -- start 2024-03-25T19:43:50.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.304+0000 7fdd00f9e640 1 -- start start 2024-03-25T19:43:50.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.304+0000 7fdd00f9e640 1 librados: setting wanted keys 2024-03-25T19:43:50.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.304+0000 7fdd00f9e640 1 librados: calling monclient init 2024-03-25T19:43:50.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.304+0000 7fdd00f9e640 10 monclient: init 2024-03-25T19:43:50.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.304+0000 7fdd00f9e640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:50.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.304+0000 7fdd00f9e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:50.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.304+0000 7fdd00f9e640 10 monclient(hunting): picked mon.noname-c con 0x7fdcfc0b3fa0 addr v1:172.21.15.44:6790/0 2024-03-25T19:43:50.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.304+0000 7fdd00f9e640 10 monclient(hunting): picked mon.noname-b con 0x7fdcfc0b49a0 addr v1:172.21.15.111:6789/0 2024-03-25T19:43:50.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.304+0000 7fdd00f9e640 10 monclient(hunting): picked mon.noname-a con 0x7fdcfc13b330 addr v1:172.21.15.44:6789/0 2024-03-25T19:43:50.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.304+0000 7fdd00f9e640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcfc13ae10 con 0x7fdcfc13b330 2024-03-25T19:43:50.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.304+0000 7fdd00f9e640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcfc141dc0 con 0x7fdcfc0b49a0 2024-03-25T19:43:50.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.305+0000 7fdd00f9e640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcfc141f50 con 0x7fdcfc0b3fa0 2024-03-25T19:43:50.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.305+0000 7fdd00f9e640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:50.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.305+0000 7fdd00f9e640 10 monclient(hunting): authenticate will time out at 5894.475844s 2024-03-25T19:43:50.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.323+0000 7fdcfbfff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fdcfc13b330 0x7fdcfc13b730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:50260/0 (socket says 172.21.15.44:50260) 2024-03-25T19:43:50.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.323+0000 7fdcfbfff640 1 -- 172.21.15.44:0/2936697462 learned_addr learned my addr 172.21.15.44:0/2936697462 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:43:50.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.325+0000 7fdce3fff640 10 client.?.objecter ms_handle_connect 0x7fdcfc13b330 2024-03-25T19:43:50.325 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.325+0000 7fdce3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:50.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.326+0000 7fdce3fff640 10 client.?.objecter ms_handle_connect 0x7fdcfc0b49a0 2024-03-25T19:43:50.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.326+0000 7fdce3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:50.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.329+0000 7fdce3fff640 1 -- 172.21.15.44:0/2936697462 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 414377185 0 0) 0x7fdcfc141dc0 con 0x7fdcfc0b49a0 2024-03-25T19:43:50.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.329+0000 7fdce3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:50.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.329+0000 7fdce3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:50.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.329+0000 7fdce3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:50.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.329+0000 7fdce3fff640 10 monclient(hunting): my global_id is 4603 2024-03-25T19:43:50.329 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.329+0000 7fdce3fff640 1 -- 172.21.15.44:0/2936697462 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdce8004eb0 con 0x7fdcfc0b49a0 2024-03-25T19:43:50.331 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.331+0000 7fdce3fff640 1 -- 172.21.15.44:0/2936697462 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942320035 0 0) 0x7fdcfc13ae10 con 0x7fdcfc13b330 2024-03-25T19:43:50.331 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.331+0000 7fdce3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:50.331 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.331+0000 7fdce3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:50.331 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.331+0000 7fdce3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:50.331 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.331+0000 7fdce3fff640 10 monclient(hunting): my global_id is 4614 2024-03-25T19:43:50.331 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.331+0000 7fdce3fff640 1 -- 172.21.15.44:0/2936697462 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcfc141dc0 con 0x7fdcfc13b330 2024-03-25T19:43:50.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.334+0000 7fdce3fff640 1 -- 172.21.15.44:0/2936697462 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1642313264 0 0) 0x7fdce8004eb0 con 0x7fdcfc0b49a0 2024-03-25T19:43:50.335 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.334+0000 7fdce3fff640 1 -- 172.21.15.44:0/2936697462 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcfc13ae10 con 0x7fdcfc0b49a0 2024-03-25T19:43:50.335 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.334+0000 7fdce3fff640 1 -- 172.21.15.44:0/2936697462 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fdcf0001550 con 0x7fdcfc0b49a0 2024-03-25T19:43:50.335 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.334+0000 7fdce3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:50.335 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.334+0000 7fdce3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:50.335 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.334+0000 7fdce3fff640 10 monclient(hunting): dump: 2024-03-25T19:43:50.335 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:50.335 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:50.335 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:50.335 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:50.335 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:50.335 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:50.335 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:50.335 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:50.335 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:50.336 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:50.336 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.334+0000 7fdce3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:50.336 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.335+0000 7fdce3fff640 10 client.?.objecter ms_handle_connect 0x7fdcfc0b3fa0 2024-03-25T19:43:50.336 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.335+0000 7fdce3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:50.337 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.337+0000 7fdce3fff640 1 -- 172.21.15.44:0/2936697462 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3027053944 0 0) 0x7fdcfc141dc0 con 0x7fdcfc13b330 2024-03-25T19:43:50.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.337+0000 7fdce3fff640 1 -- 172.21.15.44:0/2936697462 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdce8004eb0 con 0x7fdcfc13b330 2024-03-25T19:43:50.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.337+0000 7fdce3fff640 1 -- 172.21.15.44:0/2936697462 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fdcf4001d30 con 0x7fdcfc13b330 2024-03-25T19:43:50.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.337+0000 7fdce3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:50.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.337+0000 7fdce3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:50.338 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.337+0000 7fdce3fff640 10 monclient(hunting): dump: 2024-03-25T19:43:50.338 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:50.338 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:50.338 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:50.338 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:50.338 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:50.338 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:50.338 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:50.338 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:50.339 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:50.339 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:50.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.342+0000 7fdce3fff640 1 -- 172.21.15.44:0/2936697462 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3435080215 0 0) 0x7fdcfc13ae10 con 0x7fdcfc0b49a0 2024-03-25T19:43:50.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.342+0000 7fdce3fff640 1 -- 172.21.15.44:0/2936697462 >> v1:172.21.15.44:6790/0 conn(0x7fdcfc0b3fa0 legacy=0x7fdcfc1df1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:50.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.342+0000 7fdce3fff640 1 -- 172.21.15.44:0/2936697462 >> v1:172.21.15.44:6789/0 conn(0x7fdcfc13b330 legacy=0x7fdcfc13b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:50.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.342+0000 7fdce3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:43:50.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.342+0000 7fdce3fff640 1 monclient: found mon.b 2024-03-25T19:43:50.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.342+0000 7fdce3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:50.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.342+0000 7fdce3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:43:50.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.342+0000 7fdce3fff640 1 -- 172.21.15.44:0/2936697462 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcfc13be40 con 0x7fdcfc0b49a0 2024-03-25T19:43:50.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.342+0000 7fdce3fff640 10 monclient: _finish_auth 0 2024-03-25T19:43:50.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.342+0000 7fdce3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:43:50.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.342+0000 7fdce3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:50.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.342+0000 7fdd00f9e640 5 monclient: authenticate success, global_id 4603 2024-03-25T19:43:50.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.342+0000 7fdd00f9e640 10 monclient: _renew_subs 2024-03-25T19:43:50.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.342+0000 7fdd00f9e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:43:50.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.342+0000 7fdd00f9e640 1 -- 172.21.15.44:0/2936697462 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcfc1e7490 con 0x7fdcfc0b49a0 2024-03-25T19:43:50.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.342+0000 7fdd00f9e640 10 client.4603.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:50.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.342+0000 7fdd00f9e640 10 monclient: _renew_subs 2024-03-25T19:43:50.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.343+0000 7fdd00f9e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:43:50.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.343+0000 7fdd00f9e640 1 -- 172.21.15.44:0/2936697462 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcfc1e7800 con 0x7fdcfc0b49a0 2024-03-25T19:43:50.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.343+0000 7fdd00f9e640 1 librados: init done 2024-03-25T19:43:50.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.344+0000 7fdd00f9e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:50.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.344+0000 7fdd00f9e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:50.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.344+0000 7fdd00f9e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:43:50.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.344+0000 7fdd00f9e640 1 -- 172.21.15.44:0/2936697462 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcfc0627b0 con 0x7fdcfc0b49a0 2024-03-25T19:43:50.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.615+0000 7fdce3fff640 1 -- 172.21.15.44:0/2936697462 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcf0004a90 con 0x7fdcfc0b49a0 2024-03-25T19:43:50.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.615+0000 7fdce3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:50.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.615+0000 7fdce3fff640 1 -- 172.21.15.44:0/2936697462 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fdcf0004da0 con 0x7fdcfc0b49a0 2024-03-25T19:43:50.615 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.615+0000 7fdce3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:50.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.615+0000 7fdce3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:50.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.615+0000 7fdce3fff640 10 monclient: dump: 2024-03-25T19:43:50.616 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:50.616 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:50.616 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:50.616 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:50.616 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:50.616 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:50.616 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:50.616 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:50.616 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:50.616 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:50.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.615+0000 7fdcf9ffb640 4 set_mon_vals no callback set 2024-03-25T19:43:50.617 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.616+0000 7fdce3fff640 1 -- 172.21.15.44:0/2936697462 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7fdcf001ad70 con 0x7fdcfc0b49a0 2024-03-25T19:43:50.617 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.616+0000 7fdcfb7fe640 10 monclient: get_auth_request con 0x7fdce806ddf0 auth_method 0 2024-03-25T19:43:50.617 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.617+0000 7fdce3fff640 10 client.4603.objecter ms_handle_connect 0x7fdce806ddf0 2024-03-25T19:43:50.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.620+0000 7fdce3fff640 1 -- 172.21.15.44:0/2936697462 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7fdcf00548b0 con 0x7fdcfc0b49a0 2024-03-25T19:43:50.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.620+0000 7fdce3fff640 10 client.4603.objecter ms_dispatch 0x7fdcfc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:50.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.620+0000 7fdce3fff640 3 client.4603.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:50.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.620+0000 7fdce3fff640 3 client.4603.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:50.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.620+0000 7fdce3fff640 20 client.4603.objecter dump_active .. 0 homeless 2024-03-25T19:43:50.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.621+0000 7fdce3fff640 1 -- 172.21.15.44:0/2936697462 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdcf00503c0 con 0x7fdcfc0b49a0 2024-03-25T19:43:50.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.621+0000 7fdce3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:50.621 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.621+0000 7fdce3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:50.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.851+0000 7fdd00f9e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:50.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.851+0000 7fdd00f9e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:50.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.851+0000 7fdd00f9e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:43:50.856 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.851+0000 7fdd00f9e640 1 -- 172.21.15.44:0/2936697462 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcfc13afa0 con 0x7fdcfc0b49a0 2024-03-25T19:43:50.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.871+0000 7fdce3fff640 1 -- 172.21.15.44:0/2936697462 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7fdcf00522f0 con 0x7fdcfc0b49a0 2024-03-25T19:43:50.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.871+0000 7fdce3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:50.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.871+0000 7fdce3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:50.871 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:43:50.871 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:50.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.873+0000 7fdd00f9e640 10 librados: watch_flush enter 2024-03-25T19:43:50.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.873+0000 7fdd00f9e640 10 librados: watch_flush exit 2024-03-25T19:43:50.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.873+0000 7fdd00f9e640 20 client.4603.objecter shutdown clearing up homeless session... 2024-03-25T19:43:50.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.873+0000 7fdd00f9e640 10 client.4603.objecter successfully canceled tick 2024-03-25T19:43:50.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.873+0000 7fdd00f9e640 1 -- 172.21.15.44:0/2936697462 >> v1:172.21.15.111:6800/3883215793 conn(0x7fdce806ddf0 legacy=0x7fdce80702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:50.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.873+0000 7fdd00f9e640 10 monclient: shutdown 2024-03-25T19:43:50.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.873+0000 7fdd00f9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:50.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.873+0000 7fdd00f9e640 1 -- 172.21.15.44:0/2936697462 >> v1:172.21.15.111:6789/0 conn(0x7fdcfc0b49a0 legacy=0x7fdcfc1df8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:50.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.873+0000 7fdd00f9e640 1 -- 172.21.15.44:0/2936697462 shutdown_connections 2024-03-25T19:43:50.874 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.873+0000 7fdd00f9e640 1 -- 172.21.15.44:0/2936697462 >> 172.21.15.44:0/2936697462 conn(0x7fdcfc0615c0 msgr2=0x7fdcfc0b6340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:50.875 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.873+0000 7fdd00f9e640 1 -- 172.21.15.44:0/2936697462 shutdown_connections 2024-03-25T19:43:50.875 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.873+0000 7fdd00f9e640 1 -- 172.21.15.44:0/2936697462 wait complete. 2024-03-25T19:43:50.875 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:50.873+0000 7fdd00f9e640 1 librados: shutdown 2024-03-25T19:43:50.884 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:57.690 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:57.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.805+0000 7f04befcf640 10 monclient: get_monmap_and_config 2024-03-25T19:43:57.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.805+0000 7f04befcf640 10 monclient: build_initial_monmap 2024-03-25T19:43:57.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.805+0000 7f04befcf640 10 monclient: monmap: 2024-03-25T19:43:57.806 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:43:57.806 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:57.806 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:43:57.806933+0000 2024-03-25T19:43:57.806 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:43:57.806933+0000 2024-03-25T19:43:57.806 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:57.806 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:57.806 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:43:57.806 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:43:57.806 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:43:57.806 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:57.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.806+0000 7f04befcf640 1 Processor -- start 2024-03-25T19:43:57.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.806+0000 7f04befcf640 1 -- start start 2024-03-25T19:43:57.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.806+0000 7f04befcf640 10 monclient: init 2024-03-25T19:43:57.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.807+0000 7f04befcf640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:57.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.807+0000 7f04befcf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:57.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.807+0000 7f04befcf640 10 monclient(hunting): picked mon.noname-b con 0x7f04b80b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T19:43:57.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.807+0000 7f04befcf640 10 monclient(hunting): picked mon.noname-c con 0x7f04b80b1730 addr v1:172.21.15.111:6789/0 2024-03-25T19:43:57.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.807+0000 7f04befcf640 10 monclient(hunting): picked mon.noname-a con 0x7f04b814be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:43:57.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.807+0000 7f04befcf640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b80af940 con 0x7f04b814be90 2024-03-25T19:43:57.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.807+0000 7f04befcf640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b80b0a70 con 0x7f04b80b1730 2024-03-25T19:43:57.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.807+0000 7f04befcf640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b80b0d50 con 0x7f04b80b4d60 2024-03-25T19:43:57.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.807+0000 7f04befcf640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:57.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.807+0000 7f04befcf640 10 monclient(hunting): authenticate will time out at 5901.978280s 2024-03-25T19:43:57.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.818+0000 7f04bcd6b640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f04b80b4d60 0x7f04b813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:57326/0 (socket says 172.21.15.44:57326) 2024-03-25T19:43:57.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.818+0000 7f04bcd6b640 1 -- 172.21.15.44:0/4092572554 learned_addr learned my addr 172.21.15.44:0/4092572554 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:43:57.827 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.827+0000 7f04b77fe640 1 -- 172.21.15.44:0/4092572554 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439686460 0 0) 0x7f04b80af940 con 0x7f04b814be90 2024-03-25T19:43:57.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.827+0000 7f04b77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:57.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.827+0000 7f04b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:57.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.827+0000 7f04b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:57.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.827+0000 7f04b77fe640 10 monclient(hunting): my global_id is 4617 2024-03-25T19:43:57.828 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.827+0000 7f04b77fe640 1 -- 172.21.15.44:0/4092572554 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a8003520 con 0x7f04b814be90 2024-03-25T19:43:57.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.835+0000 7f04b77fe640 1 -- 172.21.15.44:0/4092572554 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 539909558 0 0) 0x7f04b80b0d50 con 0x7f04b80b4d60 2024-03-25T19:43:57.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.835+0000 7f04b77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:57.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.835+0000 7f04b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:57.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.835+0000 7f04b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:57.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.835+0000 7f04b77fe640 10 monclient(hunting): my global_id is 4595 2024-03-25T19:43:57.835 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.835+0000 7f04b77fe640 1 -- 172.21.15.44:0/4092572554 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b80af940 con 0x7f04b80b4d60 2024-03-25T19:43:57.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.850+0000 7f04b77fe640 1 -- 172.21.15.44:0/4092572554 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 942995303 0 0) 0x7f04a8003520 con 0x7f04b814be90 2024-03-25T19:43:57.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.850+0000 7f04b77fe640 1 -- 172.21.15.44:0/4092572554 >> v1:172.21.15.44:6790/0 conn(0x7f04b80b4d60 legacy=0x7f04b813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:57.850 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.850+0000 7f04b77fe640 1 -- 172.21.15.44:0/4092572554 >> v1:172.21.15.111:6789/0 conn(0x7f04b80b1730 legacy=0x7f04b80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:57.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.850+0000 7f04b77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:43:57.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.850+0000 7f04b77fe640 1 monclient: found mon.noname-a 2024-03-25T19:43:57.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.850+0000 7f04b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:57.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.850+0000 7f04b77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:43:57.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.850+0000 7f04b77fe640 1 -- 172.21.15.44:0/4092572554 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b80b1030 con 0x7f04b814be90 2024-03-25T19:43:57.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.850+0000 7f04b77fe640 10 monclient: _finish_auth 0 2024-03-25T19:43:57.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.850+0000 7f04b77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:43:57.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.850+0000 7f04b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:57.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.850+0000 7f04b77fe640 1 -- 172.21.15.44:0/4092572554 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f04b0001a10 con 0x7f04b814be90 2024-03-25T19:43:57.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.850+0000 7f04befcf640 5 monclient: authenticate success, global_id 4617 2024-03-25T19:43:57.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.850+0000 7f04b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:57.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.850+0000 7f04b77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:57.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.850+0000 7f04b77fe640 10 monclient: dump: 2024-03-25T19:43:57.851 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:57.852 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:57.852 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:57.852 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:57.852 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:57.852 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:57.852 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:57.852 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:57.852 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:57.852 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:57.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.850+0000 7f04befcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:57.865 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.865+0000 7f04b77fe640 1 -- 172.21.15.44:0/4092572554 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04b0001b70 con 0x7f04b814be90 2024-03-25T19:43:57.865 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.865+0000 7f04b77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:57.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.865+0000 7f04b77fe640 1 -- 172.21.15.44:0/4092572554 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f04b0001f80 con 0x7f04b814be90 2024-03-25T19:43:57.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.865+0000 7f04befcf640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:57.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.865+0000 7f04befcf640 4 set_mon_vals no callback set 2024-03-25T19:43:57.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.865+0000 7f04b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:57.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.865+0000 7f04b77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:57.866 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.865+0000 7f04b77fe640 10 monclient: dump: 2024-03-25T19:43:57.866 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:57.866 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:57.866 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:57.866 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:57.866 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:57.866 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:57.866 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:57.867 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:57.867 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:57.867 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:57.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.866+0000 7f04befcf640 10 monclient: shutdown 2024-03-25T19:43:57.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.866+0000 7f04befcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:57.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.866+0000 7f04befcf640 1 -- 172.21.15.44:0/4092572554 >> v1:172.21.15.44:6789/0 conn(0x7f04b814be90 legacy=0x7f04b814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:57.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.866+0000 7f04befcf640 1 -- 172.21.15.44:0/4092572554 shutdown_connections 2024-03-25T19:43:57.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.866+0000 7f04befcf640 1 -- 172.21.15.44:0/4092572554 >> 172.21.15.44:0/4092572554 conn(0x7f04b80617a0 msgr2=0x7f04b8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:57.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.866+0000 7f04befcf640 1 -- 172.21.15.44:0/4092572554 shutdown_connections 2024-03-25T19:43:57.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.866+0000 7f04befcf640 1 -- 172.21.15.44:0/4092572554 wait complete. 2024-03-25T19:43:57.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.867+0000 7f04befcf640 10 monclient: build_initial_monmap 2024-03-25T19:43:57.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.867+0000 7f04befcf640 10 monclient: monmap: 2024-03-25T19:43:57.869 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:43:57.869 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:57.869 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:43:57.870 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:43:57.870 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:57.870 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:43:57.870 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:43:57.870 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:43:57.870 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:43:57.870 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:57.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.867+0000 7f04befcf640 1 librados: starting msgr at 2024-03-25T19:43:57.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.867+0000 7f04befcf640 1 librados: starting objecter 2024-03-25T19:43:57.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.867+0000 7f04befcf640 1 Processor -- start 2024-03-25T19:43:57.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.867+0000 7f04befcf640 1 -- start start 2024-03-25T19:43:57.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.867+0000 7f04befcf640 1 librados: setting wanted keys 2024-03-25T19:43:57.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.867+0000 7f04befcf640 1 librados: calling monclient init 2024-03-25T19:43:57.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.867+0000 7f04befcf640 10 monclient: init 2024-03-25T19:43:57.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.868+0000 7f04befcf640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:57.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.868+0000 7f04befcf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:57.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.868+0000 7f04befcf640 10 monclient(hunting): picked mon.noname-a con 0x7f04b80b1730 addr v1:172.21.15.44:6789/0 2024-03-25T19:43:57.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.868+0000 7f04befcf640 10 monclient(hunting): picked mon.noname-b con 0x7f04b80b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T19:43:57.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.868+0000 7f04befcf640 10 monclient(hunting): picked mon.noname-c con 0x7f04b814be90 addr v1:172.21.15.44:6790/0 2024-03-25T19:43:57.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.868+0000 7f04befcf640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b8141da0 con 0x7f04b80b1730 2024-03-25T19:43:57.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.868+0000 7f04befcf640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b81409a0 con 0x7f04b80b4d60 2024-03-25T19:43:57.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.868+0000 7f04befcf640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b81456f0 con 0x7f04b814be90 2024-03-25T19:43:57.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.868+0000 7f04befcf640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:57.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.868+0000 7f04befcf640 10 monclient(hunting): authenticate will time out at 5902.039235s 2024-03-25T19:43:57.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.871+0000 7f04b7fff640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f04b80b4d60 0x7f04b8142b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:56544/0 (socket says 172.21.15.44:56544) 2024-03-25T19:43:57.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.871+0000 7f04b7fff640 1 -- 172.21.15.44:0/2586549897 learned_addr learned my addr 172.21.15.44:0/2586549897 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:43:57.873 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.872+0000 7f04b57fa640 10 client.?.objecter ms_handle_connect 0x7f04b80b4d60 2024-03-25T19:43:57.873 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.872+0000 7f04b57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:57.877 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.877+0000 7f04b57fa640 1 -- 172.21.15.44:0/2586549897 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4002005088 0 0) 0x7f04b81409a0 con 0x7f04b80b4d60 2024-03-25T19:43:57.877 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.877+0000 7f04b57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:57.877 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.877+0000 7f04b57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:57.877 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.877+0000 7f04b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:57.878 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.877+0000 7f04b57fa640 10 monclient(hunting): my global_id is 4609 2024-03-25T19:43:57.878 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.877+0000 7f04b57fa640 1 -- 172.21.15.44:0/2586549897 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f049c004eb0 con 0x7f04b80b4d60 2024-03-25T19:43:57.878 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.878+0000 7f04b57fa640 10 client.?.objecter ms_handle_connect 0x7f04b80b1730 2024-03-25T19:43:57.878 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.878+0000 7f04b57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:57.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.880+0000 7f04b57fa640 10 client.?.objecter ms_handle_connect 0x7f04b814be90 2024-03-25T19:43:57.880 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.880+0000 7f04b57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:57.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.887+0000 7f04b57fa640 1 -- 172.21.15.44:0/2586549897 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2110199031 0 0) 0x7f04b8141da0 con 0x7f04b80b1730 2024-03-25T19:43:57.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.887+0000 7f04b57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:57.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.887+0000 7f04b57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:57.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.887+0000 7f04b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:57.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.887+0000 7f04b57fa640 10 monclient(hunting): my global_id is 4620 2024-03-25T19:43:57.887 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.887+0000 7f04b57fa640 1 -- 172.21.15.44:0/2586549897 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b81409a0 con 0x7f04b80b1730 2024-03-25T19:43:57.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.887+0000 7f04b57fa640 1 -- 172.21.15.44:0/2586549897 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3803167271 0 0) 0x7f049c004eb0 con 0x7f04b80b4d60 2024-03-25T19:43:57.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.887+0000 7f04b57fa640 1 -- 172.21.15.44:0/2586549897 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04b8141da0 con 0x7f04b80b4d60 2024-03-25T19:43:57.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.887+0000 7f04b57fa640 1 -- 172.21.15.44:0/2586549897 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f04a4001a30 con 0x7f04b80b4d60 2024-03-25T19:43:57.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.887+0000 7f04b57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:57.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.887+0000 7f04b57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:57.888 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.887+0000 7f04b57fa640 10 monclient(hunting): dump: 2024-03-25T19:43:57.889 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:57.889 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:57.889 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:57.889 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:57.889 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:57.889 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:57.889 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:57.889 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:57.889 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:57.889 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:57.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.887+0000 7f04b57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:57.892 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.891+0000 7f04b57fa640 1 -- 172.21.15.44:0/2586549897 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 891857687 0 0) 0x7f04b8141da0 con 0x7f04b80b4d60 2024-03-25T19:43:57.892 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.891+0000 7f04b57fa640 1 -- 172.21.15.44:0/2586549897 >> v1:172.21.15.44:6790/0 conn(0x7f04b814be90 legacy=0x7f04b81e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:57.892 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.891+0000 7f04b57fa640 1 -- 172.21.15.44:0/2586549897 >> v1:172.21.15.44:6789/0 conn(0x7f04b80b1730 legacy=0x7f04b8142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:57.892 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.891+0000 7f04b57fa640 10 monclient: _finish_hunting 0 2024-03-25T19:43:57.892 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.891+0000 7f04b57fa640 1 monclient: found mon.b 2024-03-25T19:43:57.892 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.891+0000 7f04b57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:57.892 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.891+0000 7f04b57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:43:57.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.891+0000 7f04b57fa640 1 -- 172.21.15.44:0/2586549897 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b8145880 con 0x7f04b80b4d60 2024-03-25T19:43:57.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.891+0000 7f04b57fa640 10 monclient: _finish_auth 0 2024-03-25T19:43:57.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.891+0000 7f04b57fa640 10 monclient: _check_auth_tickets 2024-03-25T19:43:57.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.891+0000 7f04b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:57.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.891+0000 7f04befcf640 5 monclient: authenticate success, global_id 4609 2024-03-25T19:43:57.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.891+0000 7f04befcf640 10 monclient: _renew_subs 2024-03-25T19:43:57.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.891+0000 7f04befcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:43:57.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.891+0000 7f04befcf640 1 -- 172.21.15.44:0/2586549897 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b8137130 con 0x7f04b80b4d60 2024-03-25T19:43:57.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.892+0000 7f04befcf640 10 client.4609.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:57.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.892+0000 7f04befcf640 10 monclient: _renew_subs 2024-03-25T19:43:57.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.892+0000 7f04befcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:43:57.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.892+0000 7f04befcf640 1 -- 172.21.15.44:0/2586549897 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b8140560 con 0x7f04b80b4d60 2024-03-25T19:43:57.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.892+0000 7f04befcf640 1 librados: init done 2024-03-25T19:43:57.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.892+0000 7f04befcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:57.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.893+0000 7f04befcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:57.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.893+0000 7f04befcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:43:57.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.893+0000 7f04befcf640 1 -- 172.21.15.44:0/2586549897 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b8140810 con 0x7f04b80b4d60 2024-03-25T19:43:57.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.910+0000 7f04b57fa640 1 -- 172.21.15.44:0/2586549897 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04a4001c70 con 0x7f04b80b4d60 2024-03-25T19:43:57.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.910+0000 7f04b57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:57.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.910+0000 7f04b57fa640 1 -- 172.21.15.44:0/2586549897 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f04a4003a60 con 0x7f04b80b4d60 2024-03-25T19:43:57.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.910+0000 7f04b57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:57.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.910+0000 7f04b57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:57.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.910+0000 7f04b57fa640 10 monclient: dump: 2024-03-25T19:43:57.911 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:43:57.911 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:43:57.911 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:57.911 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:43:57.911 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:57.911 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:43:57.911 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:43:57.911 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:43:57.911 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:43:57.911 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:43:57.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.910+0000 7f04b6ffd640 4 set_mon_vals no callback set 2024-03-25T19:43:57.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.911+0000 7f04b57fa640 1 -- 172.21.15.44:0/2586549897 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f04a4003c40 con 0x7f04b80b4d60 2024-03-25T19:43:57.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.912+0000 7f04bcd6b640 10 monclient: get_auth_request con 0x7f049c06ddd0 auth_method 0 2024-03-25T19:43:57.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.912+0000 7f04b57fa640 10 client.4609.objecter ms_handle_connect 0x7f049c06ddd0 2024-03-25T19:43:57.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.914+0000 7f04b57fa640 1 -- 172.21.15.44:0/2586549897 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f04a40548a0 con 0x7f04b80b4d60 2024-03-25T19:43:57.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.914+0000 7f04b57fa640 10 client.4609.objecter ms_dispatch 0x7f04b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:57.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.914+0000 7f04b57fa640 3 client.4609.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:57.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.914+0000 7f04b57fa640 3 client.4609.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:57.914 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:57.914+0000 7f04b57fa640 20 client.4609.objecter dump_active .. 0 homeless 2024-03-25T19:43:58.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:58.160+0000 7f04b57fa640 1 -- 172.21.15.44:0/2586549897 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f04a40503b0 con 0x7f04b80b4d60 2024-03-25T19:43:58.162 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:58.160+0000 7f04b57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:58.162 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:58.160+0000 7f04b57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:58.423 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:58.422+0000 7f04befcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:58.423 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:58.422+0000 7f04befcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:58.423 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:58.422+0000 7f04befcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:43:58.423 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:58.422+0000 7f04befcf640 1 -- 172.21.15.44:0/2586549897 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04b81401f0 con 0x7f04b80b4d60 2024-03-25T19:43:58.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:58.443+0000 7f04b57fa640 1 -- 172.21.15.44:0/2586549897 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f04a40522e0 con 0x7f04b80b4d60 2024-03-25T19:43:58.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:58.443+0000 7f04b57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:58.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:58.443+0000 7f04b57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:58.443 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:43:58.444 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:58.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:58.445+0000 7f04befcf640 10 librados: watch_flush enter 2024-03-25T19:43:58.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:58.445+0000 7f04befcf640 10 librados: watch_flush exit 2024-03-25T19:43:58.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:58.445+0000 7f04befcf640 20 client.4609.objecter shutdown clearing up homeless session... 2024-03-25T19:43:58.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:58.445+0000 7f04befcf640 10 client.4609.objecter successfully canceled tick 2024-03-25T19:43:58.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:58.446+0000 7f04befcf640 1 -- 172.21.15.44:0/2586549897 >> v1:172.21.15.111:6800/3883215793 conn(0x7f049c06ddd0 legacy=0x7f049c070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:58.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:58.446+0000 7f04befcf640 10 monclient: shutdown 2024-03-25T19:43:58.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:58.446+0000 7f04befcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:58.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:58.446+0000 7f04befcf640 1 -- 172.21.15.44:0/2586549897 >> v1:172.21.15.111:6789/0 conn(0x7f04b80b4d60 legacy=0x7f04b8142b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:58.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:58.446+0000 7f04befcf640 1 -- 172.21.15.44:0/2586549897 shutdown_connections 2024-03-25T19:43:58.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:58.446+0000 7f04befcf640 1 -- 172.21.15.44:0/2586549897 >> 172.21.15.44:0/2586549897 conn(0x7f04b80617a0 msgr2=0x7f04b813c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:58.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:58.446+0000 7f04befcf640 1 -- 172.21.15.44:0/2586549897 shutdown_connections 2024-03-25T19:43:58.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:58.446+0000 7f04befcf640 1 -- 172.21.15.44:0/2586549897 wait complete. 2024-03-25T19:43:58.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:43:58.446+0000 7f04befcf640 1 librados: shutdown 2024-03-25T19:43:58.457 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:00.302 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:08:51.203 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:00.314 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:08:51.216 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:00.338 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:08:51.239 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:00.364 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:08:51.244 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:00.373 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:08:51.252 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:00.379 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:08:51.281 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:00.383 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:08:51.285 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:00.405 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:08:51.285 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:00.421 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:08:51.322 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:05.264 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:05.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.380+0000 7ff0b1043640 10 monclient: get_monmap_and_config 2024-03-25T19:44:05.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.380+0000 7ff0b1043640 10 monclient: build_initial_monmap 2024-03-25T19:44:05.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.380+0000 7ff0b1043640 10 monclient: monmap: 2024-03-25T19:44:05.381 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:44:05.381 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:05.381 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:44:05.381543+0000 2024-03-25T19:44:05.381 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:44:05.381543+0000 2024-03-25T19:44:05.381 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:05.381 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:05.381 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:44:05.381 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:44:05.381 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:44:05.381 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:05.382 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.381+0000 7ff0b1043640 1 Processor -- start 2024-03-25T19:44:05.382 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.381+0000 7ff0b1043640 1 -- start start 2024-03-25T19:44:05.383 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.381+0000 7ff0b1043640 10 monclient: init 2024-03-25T19:44:05.383 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.382+0000 7ff0b1043640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:05.383 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.382+0000 7ff0b1043640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:05.383 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.382+0000 7ff0b1043640 10 monclient(hunting): picked mon.noname-c con 0x7ff0ac0a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:44:05.383 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.382+0000 7ff0b1043640 10 monclient(hunting): picked mon.noname-a con 0x7ff0ac13f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:44:05.383 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.382+0000 7ff0b1043640 10 monclient(hunting): picked mon.noname-b con 0x7ff0ac1430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:44:05.383 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.382+0000 7ff0b1043640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0ac142520 con 0x7ff0ac13f710 2024-03-25T19:44:05.383 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.382+0000 7ff0b1043640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0ac0b28a0 con 0x7ff0ac0a98f0 2024-03-25T19:44:05.383 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.382+0000 7ff0b1043640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0ac0b2a30 con 0x7ff0ac1430c0 2024-03-25T19:44:05.383 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.382+0000 7ff0b1043640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:05.383 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.382+0000 7ff0b1043640 10 monclient(hunting): authenticate will time out at 5909.553176s 2024-03-25T19:44:05.388 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.388+0000 7ff0aad9d640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff0ac0a98f0 0x7ff0ac0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:52472/0 (socket says 172.21.15.44:52472) 2024-03-25T19:44:05.388 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.388+0000 7ff0aad9d640 1 -- 172.21.15.44:0/3047958184 learned_addr learned my addr 172.21.15.44:0/3047958184 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:44:05.397 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.397+0000 7ff0a9d9b640 1 -- 172.21.15.44:0/3047958184 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825284526 0 0) 0x7ff0ac0b28a0 con 0x7ff0ac0a98f0 2024-03-25T19:44:05.398 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.397+0000 7ff0a9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:44:05.398 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.397+0000 7ff0a9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:05.398 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.397+0000 7ff0a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:05.398 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.397+0000 7ff0a9d9b640 10 monclient(hunting): my global_id is 4612 2024-03-25T19:44:05.398 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.397+0000 7ff0a9d9b640 1 -- 172.21.15.44:0/3047958184 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff094003520 con 0x7ff0ac0a98f0 2024-03-25T19:44:05.404 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.404+0000 7ff0a9d9b640 1 -- 172.21.15.44:0/3047958184 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330567867 0 0) 0x7ff0ac0b2a30 con 0x7ff0ac1430c0 2024-03-25T19:44:05.404 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.404+0000 7ff0a9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:44:05.404 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.404+0000 7ff0a9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:05.404 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.404+0000 7ff0a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:05.404 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.404+0000 7ff0a9d9b640 10 monclient(hunting): my global_id is 4601 2024-03-25T19:44:05.404 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.404+0000 7ff0a9d9b640 1 -- 172.21.15.44:0/3047958184 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0ac0b28a0 con 0x7ff0ac1430c0 2024-03-25T19:44:05.410 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.409+0000 7ff0a9d9b640 1 -- 172.21.15.44:0/3047958184 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809951819 0 0) 0x7ff0ac142520 con 0x7ff0ac13f710 2024-03-25T19:44:05.410 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.409+0000 7ff0a9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:44:05.410 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.409+0000 7ff0a9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:05.410 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.409+0000 7ff0a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:05.410 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.409+0000 7ff0a9d9b640 10 monclient(hunting): my global_id is 4623 2024-03-25T19:44:05.410 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.409+0000 7ff0a9d9b640 1 -- 172.21.15.44:0/3047958184 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0ac0b2a30 con 0x7ff0ac13f710 2024-03-25T19:44:05.412 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.412+0000 7ff0a9d9b640 1 -- 172.21.15.44:0/3047958184 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2979476700 0 0) 0x7ff094003520 con 0x7ff0ac0a98f0 2024-03-25T19:44:05.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.412+0000 7ff0a9d9b640 1 -- 172.21.15.44:0/3047958184 >> v1:172.21.15.44:6790/0 conn(0x7ff0ac1430c0 legacy=0x7ff0ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:05.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.412+0000 7ff0a9d9b640 1 -- 172.21.15.44:0/3047958184 >> v1:172.21.15.44:6789/0 conn(0x7ff0ac13f710 legacy=0x7ff0ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:05.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.412+0000 7ff0a9d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:44:05.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.412+0000 7ff0a9d9b640 1 monclient: found mon.noname-c 2024-03-25T19:44:05.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.412+0000 7ff0a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:05.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.412+0000 7ff0a9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:44:05.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.412+0000 7ff0a9d9b640 1 -- 172.21.15.44:0/3047958184 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ac0b44c0 con 0x7ff0ac0a98f0 2024-03-25T19:44:05.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.412+0000 7ff0a9d9b640 10 monclient: _finish_auth 0 2024-03-25T19:44:05.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.412+0000 7ff0a9d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:44:05.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.412+0000 7ff0a9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:05.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.412+0000 7ff0a9d9b640 1 -- 172.21.15.44:0/3047958184 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff09c001a10 con 0x7ff0ac0a98f0 2024-03-25T19:44:05.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.412+0000 7ff0a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:05.413 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.412+0000 7ff0a9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:05.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.412+0000 7ff0a9d9b640 10 monclient: dump: 2024-03-25T19:44:05.414 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:05.414 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:05.414 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:05.414 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:05.414 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:05.414 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:05.414 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:05.414 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:05.414 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:05.414 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:05.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.412+0000 7ff0b1043640 5 monclient: authenticate success, global_id 4612 2024-03-25T19:44:05.414 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.412+0000 7ff0b1043640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:05.424 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.424+0000 7ff0a9d9b640 1 -- 172.21.15.44:0/3047958184 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff09c001b70 con 0x7ff0ac0a98f0 2024-03-25T19:44:05.424 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.424+0000 7ff0a9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:05.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.424+0000 7ff0a9d9b640 1 -- 172.21.15.44:0/3047958184 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff09c001f80 con 0x7ff0ac0a98f0 2024-03-25T19:44:05.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.424+0000 7ff0b1043640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:05.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.424+0000 7ff0b1043640 4 set_mon_vals no callback set 2024-03-25T19:44:05.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.424+0000 7ff0a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:05.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.424+0000 7ff0a9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:05.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.424+0000 7ff0a9d9b640 10 monclient: dump: 2024-03-25T19:44:05.425 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:05.425 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:05.425 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:05.425 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:05.425 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:05.425 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:05.425 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:05.425 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:05.425 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:05.426 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:05.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.425+0000 7ff0b1043640 10 monclient: shutdown 2024-03-25T19:44:05.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.425+0000 7ff0b1043640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:05.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.425+0000 7ff0b1043640 1 -- 172.21.15.44:0/3047958184 >> v1:172.21.15.111:6789/0 conn(0x7ff0ac0a98f0 legacy=0x7ff0ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:05.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.425+0000 7ff0b1043640 1 -- 172.21.15.44:0/3047958184 shutdown_connections 2024-03-25T19:44:05.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.425+0000 7ff0b1043640 1 -- 172.21.15.44:0/3047958184 >> 172.21.15.44:0/3047958184 conn(0x7ff0ac0b7ea0 msgr2=0x7ff0ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:05.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.425+0000 7ff0b1043640 1 -- 172.21.15.44:0/3047958184 shutdown_connections 2024-03-25T19:44:05.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.425+0000 7ff0b1043640 1 -- 172.21.15.44:0/3047958184 wait complete. 2024-03-25T19:44:05.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.426+0000 7ff0b1043640 10 monclient: build_initial_monmap 2024-03-25T19:44:05.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.426+0000 7ff0b1043640 10 monclient: monmap: 2024-03-25T19:44:05.428 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:44:05.428 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:05.428 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:44:05.428 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:44:05.428 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:05.428 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:44:05.428 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:44:05.428 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:44:05.428 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:44:05.428 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:05.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.426+0000 7ff0b1043640 1 librados: starting msgr at 2024-03-25T19:44:05.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.426+0000 7ff0b1043640 1 librados: starting objecter 2024-03-25T19:44:05.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.427+0000 7ff0b1043640 1 Processor -- start 2024-03-25T19:44:05.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.427+0000 7ff0b1043640 1 -- start start 2024-03-25T19:44:05.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.427+0000 7ff0b1043640 1 librados: setting wanted keys 2024-03-25T19:44:05.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.427+0000 7ff0b1043640 1 librados: calling monclient init 2024-03-25T19:44:05.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.427+0000 7ff0b1043640 10 monclient: init 2024-03-25T19:44:05.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.427+0000 7ff0b1043640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:05.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.427+0000 7ff0b1043640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:05.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.427+0000 7ff0b1043640 10 monclient(hunting): picked mon.noname-b con 0x7ff0ac0a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:44:05.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.427+0000 7ff0b1043640 10 monclient(hunting): picked mon.noname-c con 0x7ff0ac13f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:44:05.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.427+0000 7ff0b1043640 10 monclient(hunting): picked mon.noname-a con 0x7ff0ac1430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:44:05.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.428+0000 7ff0b1043640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0ac0aca40 con 0x7ff0ac1430c0 2024-03-25T19:44:05.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.428+0000 7ff0b1043640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0ac0abf10 con 0x7ff0ac0a98f0 2024-03-25T19:44:05.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.428+0000 7ff0b1043640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0ac138f60 con 0x7ff0ac13f710 2024-03-25T19:44:05.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.428+0000 7ff0b1043640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:05.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.428+0000 7ff0b1043640 10 monclient(hunting): authenticate will time out at 5909.598879s 2024-03-25T19:44:05.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.437+0000 7ff0ab59e640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7ff0ac1430c0 0x7ff0ac1e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:55038/0 (socket says 172.21.15.44:55038) 2024-03-25T19:44:05.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.437+0000 7ff0ab59e640 1 -- 172.21.15.44:0/1802147763 learned_addr learned my addr 172.21.15.44:0/1802147763 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:44:05.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.439+0000 7ff0937fe640 10 client.?.objecter ms_handle_connect 0x7ff0ac1430c0 2024-03-25T19:44:05.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.439+0000 7ff0937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:05.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.440+0000 7ff0937fe640 10 client.?.objecter ms_handle_connect 0x7ff0ac13f710 2024-03-25T19:44:05.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.440+0000 7ff0937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:05.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.444+0000 7ff0937fe640 10 client.?.objecter ms_handle_connect 0x7ff0ac0a98f0 2024-03-25T19:44:05.444 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.444+0000 7ff0937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:05.445 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.445+0000 7ff0937fe640 1 -- 172.21.15.44:0/1802147763 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798312094 0 0) 0x7ff0ac0aca40 con 0x7ff0ac1430c0 2024-03-25T19:44:05.445 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.445+0000 7ff0937fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:05.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.445+0000 7ff0937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:05.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.445+0000 7ff0937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:05.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.445+0000 7ff0937fe640 10 monclient(hunting): my global_id is 4626 2024-03-25T19:44:05.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.445+0000 7ff0937fe640 1 -- 172.21.15.44:0/1802147763 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff098004eb0 con 0x7ff0ac1430c0 2024-03-25T19:44:05.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.445+0000 7ff0937fe640 1 -- 172.21.15.44:0/1802147763 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1476727545 0 0) 0x7ff0ac138f60 con 0x7ff0ac13f710 2024-03-25T19:44:05.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.445+0000 7ff0937fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:05.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.445+0000 7ff0937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:05.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.445+0000 7ff0937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:05.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.445+0000 7ff0937fe640 10 monclient(hunting): my global_id is 4604 2024-03-25T19:44:05.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.445+0000 7ff0937fe640 1 -- 172.21.15.44:0/1802147763 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0ac0aca40 con 0x7ff0ac13f710 2024-03-25T19:44:05.451 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.451+0000 7ff0937fe640 1 -- 172.21.15.44:0/1802147763 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 559009597 0 0) 0x7ff098004eb0 con 0x7ff0ac1430c0 2024-03-25T19:44:05.452 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.451+0000 7ff0937fe640 1 -- 172.21.15.44:0/1802147763 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0ac138f60 con 0x7ff0ac1430c0 2024-03-25T19:44:05.452 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.451+0000 7ff0937fe640 1 -- 172.21.15.44:0/1802147763 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff0a4001910 con 0x7ff0ac1430c0 2024-03-25T19:44:05.452 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.451+0000 7ff0937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:05.452 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.451+0000 7ff0937fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:05.452 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.451+0000 7ff0937fe640 10 monclient(hunting): dump: 2024-03-25T19:44:05.452 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:05.452 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:05.452 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:05.452 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:05.452 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:05.452 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:05.452 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:05.452 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:05.452 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:05.453 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:05.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.451+0000 7ff0937fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:05.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.466+0000 7ff0937fe640 1 -- 172.21.15.44:0/1802147763 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142405763 0 0) 0x7ff0ac0abf10 con 0x7ff0ac0a98f0 2024-03-25T19:44:05.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.466+0000 7ff0937fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:05.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.466+0000 7ff0937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:05.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.466+0000 7ff0937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:05.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.466+0000 7ff0937fe640 10 monclient(hunting): my global_id is 4615 2024-03-25T19:44:05.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.466+0000 7ff0937fe640 1 -- 172.21.15.44:0/1802147763 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff098004eb0 con 0x7ff0ac0a98f0 2024-03-25T19:44:05.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.470+0000 7ff0937fe640 1 -- 172.21.15.44:0/1802147763 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1193775224 0 0) 0x7ff0ac138f60 con 0x7ff0ac1430c0 2024-03-25T19:44:05.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.470+0000 7ff0937fe640 1 -- 172.21.15.44:0/1802147763 >> v1:172.21.15.44:6790/0 conn(0x7ff0ac13f710 legacy=0x7ff0ac0ac330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:05.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.470+0000 7ff0937fe640 1 -- 172.21.15.44:0/1802147763 >> v1:172.21.15.111:6789/0 conn(0x7ff0ac0a98f0 legacy=0x7ff0ac138690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:05.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.470+0000 7ff0937fe640 10 monclient: _finish_hunting 0 2024-03-25T19:44:05.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.470+0000 7ff0937fe640 1 monclient: found mon.a 2024-03-25T19:44:05.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.470+0000 7ff0937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:05.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.470+0000 7ff0937fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:44:05.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.470+0000 7ff0937fe640 1 -- 172.21.15.44:0/1802147763 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ac139990 con 0x7ff0ac1430c0 2024-03-25T19:44:05.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.470+0000 7ff0937fe640 10 monclient: _finish_auth 0 2024-03-25T19:44:05.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.470+0000 7ff0937fe640 10 monclient: _check_auth_tickets 2024-03-25T19:44:05.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.470+0000 7ff0937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:05.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.470+0000 7ff0b1043640 5 monclient: authenticate success, global_id 4626 2024-03-25T19:44:05.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.470+0000 7ff0b1043640 10 monclient: _renew_subs 2024-03-25T19:44:05.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.470+0000 7ff0b1043640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:44:05.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.470+0000 7ff0b1043640 1 -- 172.21.15.44:0/1802147763 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0ac137bd0 con 0x7ff0ac1430c0 2024-03-25T19:44:05.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.470+0000 7ff0b1043640 10 client.4626.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:05.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.470+0000 7ff0b1043640 10 monclient: _renew_subs 2024-03-25T19:44:05.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.470+0000 7ff0b1043640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:44:05.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.470+0000 7ff0b1043640 1 -- 172.21.15.44:0/1802147763 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0ac1365e0 con 0x7ff0ac1430c0 2024-03-25T19:44:05.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.470+0000 7ff0b1043640 1 librados: init done 2024-03-25T19:44:05.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.471+0000 7ff0b1043640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:05.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.471+0000 7ff0b1043640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:05.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.471+0000 7ff0b1043640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:44:05.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.471+0000 7ff0b1043640 1 -- 172.21.15.44:0/1802147763 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0ac0b17e0 con 0x7ff0ac1430c0 2024-03-25T19:44:05.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.801+0000 7ff0937fe640 1 -- 172.21.15.44:0/1802147763 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0a40035f0 con 0x7ff0ac1430c0 2024-03-25T19:44:05.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.801+0000 7ff0937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:05.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.801+0000 7ff0937fe640 1 -- 172.21.15.44:0/1802147763 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff0a4003820 con 0x7ff0ac1430c0 2024-03-25T19:44:05.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.801+0000 7ff0937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:05.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.801+0000 7ff0937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:05.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.801+0000 7ff0937fe640 10 monclient: dump: 2024-03-25T19:44:05.801 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:05.801 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:05.801 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:05.801 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:05.801 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:05.801 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:05.802 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:05.802 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:05.802 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:05.802 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:05.802 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.801+0000 7ff0a959a640 4 set_mon_vals no callback set 2024-03-25T19:44:05.802 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.802+0000 7ff0937fe640 1 -- 172.21.15.44:0/1802147763 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7ff0a4003a40 con 0x7ff0ac1430c0 2024-03-25T19:44:05.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.802+0000 7ff0aad9d640 10 monclient: get_auth_request con 0x7ff09806de00 auth_method 0 2024-03-25T19:44:05.803 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.803+0000 7ff0937fe640 10 client.4626.objecter ms_handle_connect 0x7ff09806de00 2024-03-25T19:44:05.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.808+0000 7ff0937fe640 1 -- 172.21.15.44:0/1802147763 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7ff0a4054580 con 0x7ff0ac1430c0 2024-03-25T19:44:05.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.808+0000 7ff0937fe640 10 client.4626.objecter ms_dispatch 0x7ff0ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:05.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.808+0000 7ff0937fe640 3 client.4626.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:05.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.808+0000 7ff0937fe640 3 client.4626.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:05.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.808+0000 7ff0937fe640 20 client.4626.objecter dump_active .. 0 homeless 2024-03-25T19:44:05.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.808+0000 7ff0937fe640 1 -- 172.21.15.44:0/1802147763 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff0a4050090 con 0x7ff0ac1430c0 2024-03-25T19:44:05.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.808+0000 7ff0937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:05.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:05.808+0000 7ff0937fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:06.046 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:06.045+0000 7ff0b1043640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:06.046 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:06.046+0000 7ff0b1043640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:06.046 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:06.046+0000 7ff0b1043640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:44:06.046 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:06.046+0000 7ff0b1043640 1 -- 172.21.15.44:0/1802147763 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0ac138da0 con 0x7ff0ac1430c0 2024-03-25T19:44:06.068 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:06.067+0000 7ff0937fe640 1 -- 172.21.15.44:0/1802147763 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7ff0a4051fc0 con 0x7ff0ac1430c0 2024-03-25T19:44:06.068 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:06.068+0000 7ff0937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:06.068 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:06.068+0000 7ff0937fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:06.068 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:44:06.068 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:06.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:06.070+0000 7ff0b1043640 10 librados: watch_flush enter 2024-03-25T19:44:06.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:06.070+0000 7ff0b1043640 10 librados: watch_flush exit 2024-03-25T19:44:06.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:06.070+0000 7ff0b1043640 20 client.4626.objecter shutdown clearing up homeless session... 2024-03-25T19:44:06.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:06.070+0000 7ff0b1043640 10 client.4626.objecter successfully canceled tick 2024-03-25T19:44:06.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:06.070+0000 7ff0b1043640 1 -- 172.21.15.44:0/1802147763 >> v1:172.21.15.111:6800/3883215793 conn(0x7ff09806de00 legacy=0x7ff0980702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:06.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:06.070+0000 7ff0b1043640 10 monclient: shutdown 2024-03-25T19:44:06.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:06.070+0000 7ff0b1043640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:06.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:06.070+0000 7ff0b1043640 1 -- 172.21.15.44:0/1802147763 >> v1:172.21.15.44:6789/0 conn(0x7ff0ac1430c0 legacy=0x7ff0ac1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:06.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:06.070+0000 7ff0b1043640 1 -- 172.21.15.44:0/1802147763 shutdown_connections 2024-03-25T19:44:06.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:06.070+0000 7ff0b1043640 1 -- 172.21.15.44:0/1802147763 >> 172.21.15.44:0/1802147763 conn(0x7ff0ac0b7ea0 msgr2=0x7ff0ac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:06.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:06.070+0000 7ff0b1043640 1 -- 172.21.15.44:0/1802147763 shutdown_connections 2024-03-25T19:44:06.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:06.070+0000 7ff0b1043640 1 -- 172.21.15.44:0/1802147763 wait complete. 2024-03-25T19:44:06.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:06.070+0000 7ff0b1043640 1 librados: shutdown 2024-03-25T19:44:06.082 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:12.889 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:13.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.011+0000 7f401920f640 10 monclient: get_monmap_and_config 2024-03-25T19:44:13.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.011+0000 7f401920f640 10 monclient: build_initial_monmap 2024-03-25T19:44:13.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.011+0000 7f401920f640 10 monclient: monmap: 2024-03-25T19:44:13.012 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:44:13.012 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:13.012 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:44:13.012922+0000 2024-03-25T19:44:13.012 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:44:13.012922+0000 2024-03-25T19:44:13.012 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:13.012 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:13.012 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:44:13.012 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:44:13.012 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:44:13.012 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:13.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.012+0000 7f401920f640 1 Processor -- start 2024-03-25T19:44:13.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.012+0000 7f401920f640 1 -- start start 2024-03-25T19:44:13.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.012+0000 7f401920f640 10 monclient: init 2024-03-25T19:44:13.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.013+0000 7f401920f640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:13.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.013+0000 7f401920f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:44:13.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.013+0000 7f401920f640 10 monclient(hunting): picked mon.noname-b con 0x7f40140a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T19:44:13.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.013+0000 7f401920f640 10 monclient(hunting): picked mon.noname-a con 0x7f401413f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:44:13.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.013+0000 7f401920f640 10 monclient(hunting): picked mon.noname-c con 0x7f40141430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:44:13.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.013+0000 7f401920f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4014142520 con 0x7f401413f710 2024-03-25T19:44:13.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.013+0000 7f401920f640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40140b28a0 con 0x7f40141430c0 2024-03-25T19:44:13.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.013+0000 7f401920f640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40140b2a30 con 0x7f40140a98f0 2024-03-25T19:44:13.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.013+0000 7f401920f640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:13.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.013+0000 7f401920f640 10 monclient(hunting): authenticate will time out at 5917.184304s 2024-03-25T19:44:13.018 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.018+0000 7f401259c640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f401413f710 0x7f40141429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:33854/0 (socket says 172.21.15.44:33854) 2024-03-25T19:44:13.018 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.018+0000 7f401259c640 1 -- 172.21.15.44:0/147136508 learned_addr learned my addr 172.21.15.44:0/147136508 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:44:13.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.038+0000 7f4011d9b640 1 -- 172.21.15.44:0/147136508 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254251753 0 0) 0x7f4014142520 con 0x7f401413f710 2024-03-25T19:44:13.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.039+0000 7f4011d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:44:13.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.039+0000 7f4011d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:13.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.039+0000 7f4011d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:13.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.039+0000 7f4011d9b640 10 monclient(hunting): my global_id is 4629 2024-03-25T19:44:13.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.039+0000 7f4011d9b640 1 -- 172.21.15.44:0/147136508 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ffc003520 con 0x7f401413f710 2024-03-25T19:44:13.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.044+0000 7f4011d9b640 1 -- 172.21.15.44:0/147136508 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1135031402 0 0) 0x7f40140b28a0 con 0x7f40141430c0 2024-03-25T19:44:13.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.044+0000 7f4011d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:44:13.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.044+0000 7f4011d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:13.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.044+0000 7f4011d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:13.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.044+0000 7f4011d9b640 10 monclient(hunting): my global_id is 4618 2024-03-25T19:44:13.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.044+0000 7f4011d9b640 1 -- 172.21.15.44:0/147136508 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4014142520 con 0x7f40141430c0 2024-03-25T19:44:13.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.051+0000 7f4011d9b640 1 -- 172.21.15.44:0/147136508 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183224773 0 0) 0x7f40140b2a30 con 0x7f40140a98f0 2024-03-25T19:44:13.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.051+0000 7f4011d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:44:13.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.051+0000 7f4011d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:13.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.051+0000 7f4011d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:13.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.051+0000 7f4011d9b640 10 monclient(hunting): my global_id is 4607 2024-03-25T19:44:13.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.051+0000 7f4011d9b640 1 -- 172.21.15.44:0/147136508 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40140b28a0 con 0x7f40140a98f0 2024-03-25T19:44:13.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.051+0000 7f4011d9b640 1 -- 172.21.15.44:0/147136508 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4107249463 0 0) 0x7f4014142520 con 0x7f40141430c0 2024-03-25T19:44:13.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.051+0000 7f4011d9b640 1 -- 172.21.15.44:0/147136508 >> v1:172.21.15.44:6790/0 conn(0x7f40140a98f0 legacy=0x7f40140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:13.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.051+0000 7f4011d9b640 1 -- 172.21.15.44:0/147136508 >> v1:172.21.15.44:6789/0 conn(0x7f401413f710 legacy=0x7f40141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:13.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.051+0000 7f4011d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:44:13.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.051+0000 7f4011d9b640 1 monclient: found mon.noname-c 2024-03-25T19:44:13.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.052+0000 7f4011d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:13.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.052+0000 7f4011d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:44:13.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.052+0000 7f4011d9b640 1 -- 172.21.15.44:0/147136508 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40140b44c0 con 0x7f40141430c0 2024-03-25T19:44:13.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.052+0000 7f4011d9b640 10 monclient: _finish_auth 0 2024-03-25T19:44:13.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.052+0000 7f4011d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:44:13.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.052+0000 7f4011d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:13.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.052+0000 7f4011d9b640 1 -- 172.21.15.44:0/147136508 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f400c0019f0 con 0x7f40141430c0 2024-03-25T19:44:13.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.052+0000 7f4011d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:13.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.052+0000 7f4011d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:13.053 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.052+0000 7f4011d9b640 10 monclient: dump: 2024-03-25T19:44:13.053 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:13.053 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:13.054 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:13.054 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:13.054 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:13.054 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:13.054 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:13.054 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:13.054 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:13.054 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:13.054 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.052+0000 7f401920f640 5 monclient: authenticate success, global_id 4618 2024-03-25T19:44:13.054 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.052+0000 7f401920f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:13.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.060+0000 7f4011d9b640 1 -- 172.21.15.44:0/147136508 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f400c001b50 con 0x7f40141430c0 2024-03-25T19:44:13.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.060+0000 7f4011d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:13.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.060+0000 7f4011d9b640 1 -- 172.21.15.44:0/147136508 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f400c001f60 con 0x7f40141430c0 2024-03-25T19:44:13.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.060+0000 7f401920f640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:13.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.060+0000 7f401920f640 4 set_mon_vals no callback set 2024-03-25T19:44:13.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.060+0000 7f4011d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:13.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.060+0000 7f4011d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:13.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.060+0000 7f4011d9b640 10 monclient: dump: 2024-03-25T19:44:13.060 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:13.060 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:13.060 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:13.060 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:13.060 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:13.061 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:13.061 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:13.061 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:13.061 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:13.061 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:13.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.060+0000 7f401920f640 10 monclient: shutdown 2024-03-25T19:44:13.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.060+0000 7f401920f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:13.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.060+0000 7f401920f640 1 -- 172.21.15.44:0/147136508 >> v1:172.21.15.111:6789/0 conn(0x7f40141430c0 legacy=0x7f401414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:13.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.060+0000 7f401920f640 1 -- 172.21.15.44:0/147136508 shutdown_connections 2024-03-25T19:44:13.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.061+0000 7f401920f640 1 -- 172.21.15.44:0/147136508 >> 172.21.15.44:0/147136508 conn(0x7f40140b7ea0 msgr2=0x7f40140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:13.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.061+0000 7f401920f640 1 -- 172.21.15.44:0/147136508 shutdown_connections 2024-03-25T19:44:13.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.061+0000 7f401920f640 1 -- 172.21.15.44:0/147136508 wait complete. 2024-03-25T19:44:13.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.062+0000 7f401920f640 10 monclient: build_initial_monmap 2024-03-25T19:44:13.064 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.062+0000 7f401920f640 10 monclient: monmap: 2024-03-25T19:44:13.064 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:44:13.064 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:13.064 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:44:13.064 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:44:13.064 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:13.064 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:44:13.064 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:44:13.064 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:44:13.064 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:44:13.064 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:13.064 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.062+0000 7f401920f640 1 librados: starting msgr at 2024-03-25T19:44:13.064 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.062+0000 7f401920f640 1 librados: starting objecter 2024-03-25T19:44:13.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.062+0000 7f401920f640 1 Processor -- start 2024-03-25T19:44:13.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.062+0000 7f401920f640 1 -- start start 2024-03-25T19:44:13.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.062+0000 7f401920f640 1 librados: setting wanted keys 2024-03-25T19:44:13.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.062+0000 7f401920f640 1 librados: calling monclient init 2024-03-25T19:44:13.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.062+0000 7f401920f640 10 monclient: init 2024-03-25T19:44:13.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.063+0000 7f401920f640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:13.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.063+0000 7f401920f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:13.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.063+0000 7f401920f640 10 monclient(hunting): picked mon.noname-a con 0x7f40140a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:44:13.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.063+0000 7f401920f640 10 monclient(hunting): picked mon.noname-c con 0x7f401413f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:44:13.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.063+0000 7f401920f640 10 monclient(hunting): picked mon.noname-b con 0x7f40141430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:44:13.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.063+0000 7f401920f640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40141388a0 con 0x7f40140a98f0 2024-03-25T19:44:13.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.063+0000 7f401920f640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40141349e0 con 0x7f40141430c0 2024-03-25T19:44:13.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.063+0000 7f401920f640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4014138d30 con 0x7f401413f710 2024-03-25T19:44:13.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.063+0000 7f401920f640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:13.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.063+0000 7f401920f640 10 monclient(hunting): authenticate will time out at 5917.234256s 2024-03-25T19:44:13.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.070+0000 7f4012d9d640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f40140a98f0 0x7f40141da880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:33884/0 (socket says 172.21.15.44:33884) 2024-03-25T19:44:13.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.070+0000 7f4012d9d640 1 -- 172.21.15.44:0/3635754486 learned_addr learned my addr 172.21.15.44:0/3635754486 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:44:13.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.071+0000 7f3ffb7fe640 10 client.?.objecter ms_handle_connect 0x7f40140a98f0 2024-03-25T19:44:13.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.071+0000 7f3ffb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:13.076 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.076+0000 7f3ffb7fe640 1 -- 172.21.15.44:0/3635754486 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975367687 0 0) 0x7f40141388a0 con 0x7f40140a98f0 2024-03-25T19:44:13.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.076+0000 7f3ffb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:13.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.076+0000 7f3ffb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:13.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.076+0000 7f3ffb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:13.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.076+0000 7f3ffb7fe640 10 monclient(hunting): my global_id is 4632 2024-03-25T19:44:13.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.076+0000 7f3ffb7fe640 1 -- 172.21.15.44:0/3635754486 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4000004eb0 con 0x7f40140a98f0 2024-03-25T19:44:13.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.077+0000 7f3ffb7fe640 10 client.?.objecter ms_handle_connect 0x7f40141430c0 2024-03-25T19:44:13.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.077+0000 7f3ffb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:13.080 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.080+0000 7f3ffb7fe640 10 client.?.objecter ms_handle_connect 0x7f401413f710 2024-03-25T19:44:13.080 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.080+0000 7f3ffb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:13.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.085+0000 7f3ffb7fe640 1 -- 172.21.15.44:0/3635754486 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 504533583 0 0) 0x7f40141349e0 con 0x7f40141430c0 2024-03-25T19:44:13.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.085+0000 7f3ffb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:13.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.085+0000 7f3ffb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:13.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.085+0000 7f3ffb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:13.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.085+0000 7f3ffb7fe640 10 monclient(hunting): my global_id is 4621 2024-03-25T19:44:13.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.085+0000 7f3ffb7fe640 1 -- 172.21.15.44:0/3635754486 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40141388a0 con 0x7f40141430c0 2024-03-25T19:44:13.092 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.092+0000 7f3ffb7fe640 1 -- 172.21.15.44:0/3635754486 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1344583060 0 0) 0x7f4014138d30 con 0x7f401413f710 2024-03-25T19:44:13.092 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.092+0000 7f3ffb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:13.092 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.092+0000 7f3ffb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:13.092 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.092+0000 7f3ffb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:13.092 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.092+0000 7f3ffb7fe640 10 monclient(hunting): my global_id is 4610 2024-03-25T19:44:13.092 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.092+0000 7f3ffb7fe640 1 -- 172.21.15.44:0/3635754486 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40141349e0 con 0x7f401413f710 2024-03-25T19:44:13.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.097+0000 7f3ffb7fe640 1 -- 172.21.15.44:0/3635754486 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2713503999 0 0) 0x7f40141388a0 con 0x7f40141430c0 2024-03-25T19:44:13.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.097+0000 7f3ffb7fe640 1 -- 172.21.15.44:0/3635754486 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4014138d30 con 0x7f40141430c0 2024-03-25T19:44:13.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.097+0000 7f3ffb7fe640 1 -- 172.21.15.44:0/3635754486 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f400c001e60 con 0x7f40141430c0 2024-03-25T19:44:13.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.097+0000 7f3ffb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:13.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.097+0000 7f3ffb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:13.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.097+0000 7f3ffb7fe640 10 monclient(hunting): dump: 2024-03-25T19:44:13.097 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:13.098 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:13.098 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:13.098 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:13.098 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:13.098 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:13.098 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:13.098 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:13.098 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:13.098 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:13.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.097+0000 7f3ffb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:13.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.104+0000 7f3ffb7fe640 1 -- 172.21.15.44:0/3635754486 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2335817137 0 0) 0x7f4000004eb0 con 0x7f40140a98f0 2024-03-25T19:44:13.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.104+0000 7f3ffb7fe640 1 -- 172.21.15.44:0/3635754486 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40141388a0 con 0x7f40140a98f0 2024-03-25T19:44:13.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.104+0000 7f3ffb7fe640 1 -- 172.21.15.44:0/3635754486 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f4004001a10 con 0x7f40140a98f0 2024-03-25T19:44:13.104 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.104+0000 7f3ffb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:13.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.104+0000 7f3ffb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:13.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.104+0000 7f3ffb7fe640 10 monclient(hunting): dump: 2024-03-25T19:44:13.105 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:13.105 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:13.105 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:13.105 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:13.105 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:13.105 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:13.105 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:13.105 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:13.105 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:13.105 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:13.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.108+0000 7f3ffb7fe640 1 -- 172.21.15.44:0/3635754486 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 897287610 0 0) 0x7f4014138d30 con 0x7f40141430c0 2024-03-25T19:44:13.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.108+0000 7f3ffb7fe640 1 -- 172.21.15.44:0/3635754486 >> v1:172.21.15.44:6790/0 conn(0x7f401413f710 legacy=0x7f4014133bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:13.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.108+0000 7f3ffb7fe640 1 -- 172.21.15.44:0/3635754486 >> v1:172.21.15.44:6789/0 conn(0x7f40140a98f0 legacy=0x7f40141da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:13.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.108+0000 7f3ffb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:44:13.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.108+0000 7f3ffb7fe640 1 monclient: found mon.b 2024-03-25T19:44:13.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.108+0000 7f3ffb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:13.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.108+0000 7f3ffb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:44:13.109 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.108+0000 7f3ffb7fe640 1 -- 172.21.15.44:0/3635754486 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40140aba50 con 0x7f40141430c0 2024-03-25T19:44:13.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.108+0000 7f3ffb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:44:13.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.108+0000 7f3ffb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:44:13.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.108+0000 7f3ffb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:13.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.108+0000 7f401920f640 5 monclient: authenticate success, global_id 4621 2024-03-25T19:44:13.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.108+0000 7f401920f640 10 monclient: _renew_subs 2024-03-25T19:44:13.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.108+0000 7f401920f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:44:13.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.108+0000 7f401920f640 1 -- 172.21.15.44:0/3635754486 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40140b1770 con 0x7f40141430c0 2024-03-25T19:44:13.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.108+0000 7f401920f640 10 client.4621.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:13.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.108+0000 7f401920f640 10 monclient: _renew_subs 2024-03-25T19:44:13.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.108+0000 7f401920f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:44:13.110 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.108+0000 7f401920f640 1 -- 172.21.15.44:0/3635754486 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4014137ba0 con 0x7f40141430c0 2024-03-25T19:44:13.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.108+0000 7f401920f640 1 librados: init done 2024-03-25T19:44:13.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.109+0000 7f401920f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:13.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.109+0000 7f401920f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:13.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.109+0000 7f401920f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:44:13.111 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.109+0000 7f401920f640 1 -- 172.21.15.44:0/3635754486 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40140627b0 con 0x7f40141430c0 2024-03-25T19:44:13.381 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.381+0000 7f3ffb7fe640 1 -- 172.21.15.44:0/3635754486 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f400c002040 con 0x7f40141430c0 2024-03-25T19:44:13.382 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.381+0000 7f3ffb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:13.382 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.381+0000 7f3ffb7fe640 1 -- 172.21.15.44:0/3635754486 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f400c003dc0 con 0x7f40141430c0 2024-03-25T19:44:13.382 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.381+0000 7f3ffb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:13.382 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.381+0000 7f3ffb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:13.382 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.381+0000 7f3ffb7fe640 10 monclient: dump: 2024-03-25T19:44:13.382 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:13.382 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:13.382 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:13.382 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:13.382 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:13.382 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:13.382 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:13.382 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:13.383 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:13.383 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:13.383 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.381+0000 7f401159a640 4 set_mon_vals no callback set 2024-03-25T19:44:13.383 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.382+0000 7f3ffb7fe640 1 -- 172.21.15.44:0/3635754486 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f400c019d90 con 0x7f40141430c0 2024-03-25T19:44:13.383 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.383+0000 7f4012d9d640 10 monclient: get_auth_request con 0x7f400006df20 auth_method 0 2024-03-25T19:44:13.384 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.383+0000 7f3ffb7fe640 10 client.4621.objecter ms_handle_connect 0x7f400006df20 2024-03-25T19:44:13.388 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.388+0000 7f3ffb7fe640 1 -- 172.21.15.44:0/3635754486 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f400c053900 con 0x7f40141430c0 2024-03-25T19:44:13.388 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.388+0000 7f3ffb7fe640 10 client.4621.objecter ms_dispatch 0x7f4014000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:13.388 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.388+0000 7f3ffb7fe640 3 client.4621.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:13.389 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.388+0000 7f3ffb7fe640 3 client.4621.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:13.389 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.388+0000 7f3ffb7fe640 20 client.4621.objecter dump_active .. 0 homeless 2024-03-25T19:44:13.389 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.389+0000 7f3ffb7fe640 1 -- 172.21.15.44:0/3635754486 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f400c04f410 con 0x7f40141430c0 2024-03-25T19:44:13.389 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.389+0000 7f3ffb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:13.389 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.389+0000 7f3ffb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:13.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.632+0000 7f401920f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:13.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.632+0000 7f401920f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:13.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.632+0000 7f401920f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:44:13.633 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.632+0000 7f401920f640 1 -- 172.21.15.44:0/3635754486 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4014138a50 con 0x7f40141430c0 2024-03-25T19:44:13.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.691+0000 7f3ffb7fe640 1 -- 172.21.15.44:0/3635754486 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f4014138a50 con 0x7f40141430c0 2024-03-25T19:44:13.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.691+0000 7f3ffb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:13.692 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.691+0000 7f3ffb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:13.692 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:44:13.692 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:13.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.693+0000 7f401920f640 10 librados: watch_flush enter 2024-03-25T19:44:13.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.693+0000 7f401920f640 10 librados: watch_flush exit 2024-03-25T19:44:13.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.693+0000 7f401920f640 20 client.4621.objecter shutdown clearing up homeless session... 2024-03-25T19:44:13.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.693+0000 7f401920f640 10 client.4621.objecter successfully canceled tick 2024-03-25T19:44:13.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.693+0000 7f401920f640 1 -- 172.21.15.44:0/3635754486 >> v1:172.21.15.111:6800/3883215793 conn(0x7f400006df20 legacy=0x7f40000703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:13.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.693+0000 7f401920f640 10 monclient: shutdown 2024-03-25T19:44:13.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.693+0000 7f401920f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:13.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.693+0000 7f401920f640 1 -- 172.21.15.44:0/3635754486 >> v1:172.21.15.111:6789/0 conn(0x7f40141430c0 legacy=0x7f40141342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:13.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.693+0000 7f401920f640 1 -- 172.21.15.44:0/3635754486 shutdown_connections 2024-03-25T19:44:13.694 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.693+0000 7f401920f640 1 -- 172.21.15.44:0/3635754486 >> 172.21.15.44:0/3635754486 conn(0x7f40140b7ea0 msgr2=0x7f4014074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:13.695 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.693+0000 7f401920f640 1 -- 172.21.15.44:0/3635754486 shutdown_connections 2024-03-25T19:44:13.695 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.694+0000 7f401920f640 1 -- 172.21.15.44:0/3635754486 wait complete. 2024-03-25T19:44:13.695 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:13.694+0000 7f401920f640 1 librados: shutdown 2024-03-25T19:44:13.706 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:20.349 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:09:11.250 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:20.359 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:09:11.260 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:20.388 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:09:11.289 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:20.409 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:09:11.288 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:20.411 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:09:11.290 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:20.427 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:09:11.329 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:20.433 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:09:11.335 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:20.443 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:09:11.323 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:20.472 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:09:11.374 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:20.513 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:20.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.621+0000 7eff19bba640 10 monclient: get_monmap_and_config 2024-03-25T19:44:20.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.621+0000 7eff19bba640 10 monclient: build_initial_monmap 2024-03-25T19:44:20.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.622+0000 7eff19bba640 10 monclient: monmap: 2024-03-25T19:44:20.622 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:44:20.622 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:20.622 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:44:20.623128+0000 2024-03-25T19:44:20.622 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:44:20.623128+0000 2024-03-25T19:44:20.622 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:20.622 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:20.622 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:44:20.623 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:44:20.623 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:44:20.623 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:20.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.622+0000 7eff19bba640 1 Processor -- start 2024-03-25T19:44:20.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.622+0000 7eff19bba640 1 -- start start 2024-03-25T19:44:20.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.622+0000 7eff19bba640 10 monclient: init 2024-03-25T19:44:20.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.623+0000 7eff19bba640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:20.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.623+0000 7eff19bba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:44:20.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.623+0000 7eff19bba640 10 monclient(hunting): picked mon.noname-b con 0x7eff140a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T19:44:20.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.623+0000 7eff19bba640 10 monclient(hunting): picked mon.noname-a con 0x7eff1413f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:44:20.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.623+0000 7eff19bba640 10 monclient(hunting): picked mon.noname-c con 0x7eff141430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:44:20.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.623+0000 7eff19bba640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff14142520 con 0x7eff1413f710 2024-03-25T19:44:20.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.623+0000 7eff19bba640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff140b28a0 con 0x7eff141430c0 2024-03-25T19:44:20.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.623+0000 7eff19bba640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff140b2a30 con 0x7eff140a98f0 2024-03-25T19:44:20.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.623+0000 7eff19bba640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:20.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.623+0000 7eff19bba640 10 monclient(hunting): authenticate will time out at 5924.794258s 2024-03-25T19:44:20.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.623+0000 7eff13fff640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7eff141430c0 0x7eff1414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:41184/0 (socket says 172.21.15.44:41184) 2024-03-25T19:44:20.625 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.623+0000 7eff13fff640 1 -- 172.21.15.44:0/4137379457 learned_addr learned my addr 172.21.15.44:0/4137379457 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:44:20.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.722+0000 7eff127fc640 1 -- 172.21.15.44:0/4137379457 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392487194 0 0) 0x7eff14142520 con 0x7eff1413f710 2024-03-25T19:44:20.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.722+0000 7eff127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:20.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.722+0000 7eff127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:20.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.722+0000 7eff127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:20.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.722+0000 7eff127fc640 10 monclient(hunting): my global_id is 4635 2024-03-25T19:44:20.723 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.722+0000 7eff127fc640 1 -- 172.21.15.44:0/4137379457 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efefc003520 con 0x7eff1413f710 2024-03-25T19:44:20.731 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.731+0000 7eff127fc640 1 -- 172.21.15.44:0/4137379457 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2138511332 0 0) 0x7efefc003520 con 0x7eff1413f710 2024-03-25T19:44:20.732 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.731+0000 7eff127fc640 1 -- 172.21.15.44:0/4137379457 >> v1:172.21.15.44:6790/0 conn(0x7eff140a98f0 legacy=0x7eff140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:20.732 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.731+0000 7eff127fc640 1 -- 172.21.15.44:0/4137379457 >> v1:172.21.15.111:6789/0 conn(0x7eff141430c0 legacy=0x7eff1414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:20.732 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.731+0000 7eff127fc640 10 monclient: _finish_hunting 0 2024-03-25T19:44:20.732 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.731+0000 7eff127fc640 1 monclient: found mon.noname-a 2024-03-25T19:44:20.732 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.731+0000 7eff127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:20.732 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.731+0000 7eff127fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:44:20.732 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.731+0000 7eff127fc640 1 -- 172.21.15.44:0/4137379457 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff140b44c0 con 0x7eff1413f710 2024-03-25T19:44:20.732 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.731+0000 7eff127fc640 10 monclient: _finish_auth 0 2024-03-25T19:44:20.732 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.731+0000 7eff127fc640 10 monclient: _check_auth_tickets 2024-03-25T19:44:20.732 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.731+0000 7eff127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:20.732 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.731+0000 7eff19bba640 5 monclient: authenticate success, global_id 4635 2024-03-25T19:44:20.732 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.731+0000 7eff19bba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:20.739 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.738+0000 7eff127fc640 1 -- 172.21.15.44:0/4137379457 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7eff08001690 con 0x7eff1413f710 2024-03-25T19:44:20.739 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.738+0000 7eff127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:20.739 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.738+0000 7eff127fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:20.739 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.738+0000 7eff127fc640 10 monclient: dump: 2024-03-25T19:44:20.739 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:20.739 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:20.739 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:20.739 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:20.739 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:20.739 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:20.739 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:20.739 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:20.739 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:20.740 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:20.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.738+0000 7eff19bba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:20.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.750+0000 7eff127fc640 1 -- 172.21.15.44:0/4137379457 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff08001920 con 0x7eff1413f710 2024-03-25T19:44:20.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.750+0000 7eff127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:20.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.750+0000 7eff127fc640 1 -- 172.21.15.44:0/4137379457 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7eff08001c30 con 0x7eff1413f710 2024-03-25T19:44:20.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.750+0000 7eff19bba640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:20.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.750+0000 7eff19bba640 4 set_mon_vals no callback set 2024-03-25T19:44:20.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.750+0000 7eff127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:20.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.750+0000 7eff127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:20.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.750+0000 7eff127fc640 10 monclient: dump: 2024-03-25T19:44:20.751 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:20.751 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:20.752 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:20.752 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:20.752 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:20.752 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:20.752 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:20.752 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:20.752 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:20.752 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:20.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.751+0000 7eff19bba640 10 monclient: shutdown 2024-03-25T19:44:20.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.751+0000 7eff19bba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:20.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.751+0000 7eff19bba640 1 -- 172.21.15.44:0/4137379457 >> v1:172.21.15.44:6789/0 conn(0x7eff1413f710 legacy=0x7eff141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:20.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.751+0000 7eff19bba640 1 -- 172.21.15.44:0/4137379457 shutdown_connections 2024-03-25T19:44:20.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.751+0000 7eff19bba640 1 -- 172.21.15.44:0/4137379457 >> 172.21.15.44:0/4137379457 conn(0x7eff140b7ea0 msgr2=0x7eff140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:20.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.751+0000 7eff19bba640 1 -- 172.21.15.44:0/4137379457 shutdown_connections 2024-03-25T19:44:20.753 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.751+0000 7eff19bba640 1 -- 172.21.15.44:0/4137379457 wait complete. 2024-03-25T19:44:20.754 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.751+0000 7eff19bba640 10 monclient: build_initial_monmap 2024-03-25T19:44:20.754 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.751+0000 7eff19bba640 10 monclient: monmap: 2024-03-25T19:44:20.754 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:44:20.754 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:20.754 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:44:20.754 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:44:20.754 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:20.754 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:44:20.755 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:44:20.755 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:44:20.755 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:44:20.755 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:20.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.752+0000 7eff19bba640 1 librados: starting msgr at 2024-03-25T19:44:20.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.752+0000 7eff19bba640 1 librados: starting objecter 2024-03-25T19:44:20.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.752+0000 7eff19bba640 1 Processor -- start 2024-03-25T19:44:20.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.752+0000 7eff19bba640 1 -- start start 2024-03-25T19:44:20.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.752+0000 7eff19bba640 1 librados: setting wanted keys 2024-03-25T19:44:20.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.752+0000 7eff19bba640 1 librados: calling monclient init 2024-03-25T19:44:20.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.752+0000 7eff19bba640 10 monclient: init 2024-03-25T19:44:20.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.752+0000 7eff19bba640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:20.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.752+0000 7eff19bba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:20.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.752+0000 7eff19bba640 10 monclient(hunting): picked mon.noname-b con 0x7eff140a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:44:20.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.752+0000 7eff19bba640 10 monclient(hunting): picked mon.noname-c con 0x7eff1413f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:44:20.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.752+0000 7eff19bba640 10 monclient(hunting): picked mon.noname-a con 0x7eff141430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:44:20.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.753+0000 7eff19bba640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff141d7d10 con 0x7eff141430c0 2024-03-25T19:44:20.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.753+0000 7eff19bba640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff141d7ea0 con 0x7eff140a98f0 2024-03-25T19:44:20.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.753+0000 7eff19bba640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff1413b730 con 0x7eff1413f710 2024-03-25T19:44:20.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.753+0000 7eff19bba640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:20.756 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.753+0000 7eff19bba640 10 monclient(hunting): authenticate will time out at 5924.923915s 2024-03-25T19:44:20.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.771+0000 7eff13fff640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7eff141430c0 0x7eff14135730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:35750/0 (socket says 172.21.15.44:35750) 2024-03-25T19:44:20.771 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.771+0000 7eff13fff640 1 -- 172.21.15.44:0/1346848785 learned_addr learned my addr 172.21.15.44:0/1346848785 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:44:20.774 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.774+0000 7efefbfff640 10 client.?.objecter ms_handle_connect 0x7eff141430c0 2024-03-25T19:44:20.774 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.774+0000 7efefbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:20.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.775+0000 7efefbfff640 10 client.?.objecter ms_handle_connect 0x7eff1413f710 2024-03-25T19:44:20.775 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.775+0000 7efefbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:20.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.781+0000 7efefbfff640 1 -- 172.21.15.44:0/1346848785 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226572801 0 0) 0x7eff1413b730 con 0x7eff1413f710 2024-03-25T19:44:20.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.781+0000 7efefbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:20.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.781+0000 7efefbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:20.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.781+0000 7efefbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:20.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.781+0000 7efefbfff640 10 monclient(hunting): my global_id is 4616 2024-03-25T19:44:20.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.781+0000 7efefbfff640 1 -- 172.21.15.44:0/1346848785 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff00004eb0 con 0x7eff1413f710 2024-03-25T19:44:20.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.782+0000 7efefbfff640 10 client.?.objecter ms_handle_connect 0x7eff140a98f0 2024-03-25T19:44:20.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.782+0000 7efefbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:20.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.786+0000 7efefbfff640 1 -- 172.21.15.44:0/1346848785 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1386384116 0 0) 0x7eff141d7ea0 con 0x7eff140a98f0 2024-03-25T19:44:20.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.786+0000 7efefbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:20.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.786+0000 7efefbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:20.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.786+0000 7efefbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:20.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.786+0000 7efefbfff640 10 monclient(hunting): my global_id is 4627 2024-03-25T19:44:20.787 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.786+0000 7efefbfff640 1 -- 172.21.15.44:0/1346848785 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff1413b730 con 0x7eff140a98f0 2024-03-25T19:44:20.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.792+0000 7efefbfff640 1 -- 172.21.15.44:0/1346848785 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973484331 0 0) 0x7eff141d7d10 con 0x7eff141430c0 2024-03-25T19:44:20.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.792+0000 7efefbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:20.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.792+0000 7efefbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:20.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.792+0000 7efefbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:20.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.792+0000 7efefbfff640 10 monclient(hunting): my global_id is 4638 2024-03-25T19:44:20.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.792+0000 7efefbfff640 1 -- 172.21.15.44:0/1346848785 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff141d7ea0 con 0x7eff141430c0 2024-03-25T19:44:20.799 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.799+0000 7efefbfff640 1 -- 172.21.15.44:0/1346848785 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3544416589 0 0) 0x7eff00004eb0 con 0x7eff1413f710 2024-03-25T19:44:20.799 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.799+0000 7efefbfff640 1 -- 172.21.15.44:0/1346848785 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff141d7d10 con 0x7eff1413f710 2024-03-25T19:44:20.799 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.799+0000 7efefbfff640 1 -- 172.21.15.44:0/1346848785 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7eff08001650 con 0x7eff1413f710 2024-03-25T19:44:20.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.799+0000 7efefbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:20.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.799+0000 7efefbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:20.800 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.799+0000 7efefbfff640 10 monclient(hunting): dump: 2024-03-25T19:44:20.800 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:20.800 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:20.800 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:20.800 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:20.800 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:20.800 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:20.800 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:20.800 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:20.800 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:20.801 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:20.801 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.799+0000 7efefbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:20.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.805+0000 7efefbfff640 1 -- 172.21.15.44:0/1346848785 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1864756104 0 0) 0x7eff1413b730 con 0x7eff140a98f0 2024-03-25T19:44:20.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.805+0000 7efefbfff640 1 -- 172.21.15.44:0/1346848785 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff00004eb0 con 0x7eff140a98f0 2024-03-25T19:44:20.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.805+0000 7efefbfff640 1 -- 172.21.15.44:0/1346848785 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7eff040018a0 con 0x7eff140a98f0 2024-03-25T19:44:20.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.805+0000 7efefbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:20.805 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.805+0000 7efefbfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:20.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.805+0000 7efefbfff640 10 monclient(hunting): dump: 2024-03-25T19:44:20.806 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:20.806 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:20.806 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:20.806 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:20.806 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:20.806 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:20.806 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:20.806 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:20.806 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:20.806 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:20.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.818+0000 7efefbfff640 1 -- 172.21.15.44:0/1346848785 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1606523735 0 0) 0x7eff141d7d10 con 0x7eff1413f710 2024-03-25T19:44:20.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.818+0000 7efefbfff640 1 -- 172.21.15.44:0/1346848785 >> v1:172.21.15.111:6789/0 conn(0x7eff140a98f0 legacy=0x7eff141d7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:20.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.818+0000 7efefbfff640 1 -- 172.21.15.44:0/1346848785 >> v1:172.21.15.44:6789/0 conn(0x7eff141430c0 legacy=0x7eff14135730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:20.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.818+0000 7efefbfff640 10 monclient: _finish_hunting 0 2024-03-25T19:44:20.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.818+0000 7efefbfff640 1 monclient: found mon.c 2024-03-25T19:44:20.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.818+0000 7efefbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:20.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.818+0000 7efefbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:44:20.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.818+0000 7efefbfff640 1 -- 172.21.15.44:0/1346848785 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff1413b8c0 con 0x7eff1413f710 2024-03-25T19:44:20.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.818+0000 7efefbfff640 10 monclient: _finish_auth 0 2024-03-25T19:44:20.819 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.818+0000 7efefbfff640 10 monclient: _check_auth_tickets 2024-03-25T19:44:20.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.818+0000 7efefbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:20.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.818+0000 7eff19bba640 5 monclient: authenticate success, global_id 4616 2024-03-25T19:44:20.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.818+0000 7eff19bba640 10 monclient: _renew_subs 2024-03-25T19:44:20.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.818+0000 7eff19bba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:44:20.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.818+0000 7eff19bba640 1 -- 172.21.15.44:0/1346848785 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff141e3050 con 0x7eff1413f710 2024-03-25T19:44:20.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.818+0000 7eff19bba640 10 client.4616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:20.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.818+0000 7eff19bba640 10 monclient: _renew_subs 2024-03-25T19:44:20.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.818+0000 7eff19bba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:44:20.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.818+0000 7eff19bba640 1 -- 172.21.15.44:0/1346848785 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7eff141e33c0 con 0x7eff1413f710 2024-03-25T19:44:20.820 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.818+0000 7eff19bba640 1 librados: init done 2024-03-25T19:44:20.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.819+0000 7eff19bba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:20.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.819+0000 7eff19bba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:20.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.819+0000 7eff19bba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:44:20.821 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:20.819+0000 7eff19bba640 1 -- 172.21.15.44:0/1346848785 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff1413bab0 con 0x7eff1413f710 2024-03-25T19:44:21.154 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.153+0000 7efefbfff640 1 -- 172.21.15.44:0/1346848785 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff08003960 con 0x7eff1413f710 2024-03-25T19:44:21.154 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.153+0000 7efefbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:21.154 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.153+0000 7efefbfff640 1 -- 172.21.15.44:0/1346848785 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7eff08001250 con 0x7eff1413f710 2024-03-25T19:44:21.154 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.153+0000 7efefbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:21.154 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.153+0000 7efefbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:21.154 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.153+0000 7efefbfff640 10 monclient: dump: 2024-03-25T19:44:21.155 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:21.155 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:21.155 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:21.155 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:21.155 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:21.155 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:21.155 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:21.155 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:21.155 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:21.155 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:21.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.153+0000 7eff11ffb640 4 set_mon_vals no callback set 2024-03-25T19:44:21.155 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.154+0000 7efefbfff640 1 -- 172.21.15.44:0/1346848785 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7eff08019b60 con 0x7eff1413f710 2024-03-25T19:44:21.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.155+0000 7eff137fe640 10 monclient: get_auth_request con 0x7eff0006df20 auth_method 0 2024-03-25T19:44:21.156 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.155+0000 7efefbfff640 10 client.4616.objecter ms_handle_connect 0x7eff0006df20 2024-03-25T19:44:21.160 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.160+0000 7efefbfff640 1 -- 172.21.15.44:0/1346848785 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7eff08053740 con 0x7eff1413f710 2024-03-25T19:44:21.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.160+0000 7efefbfff640 10 client.4616.objecter ms_dispatch 0x7eff14000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:21.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.160+0000 7efefbfff640 3 client.4616.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:21.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.160+0000 7efefbfff640 3 client.4616.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:21.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.160+0000 7efefbfff640 20 client.4616.objecter dump_active .. 0 homeless 2024-03-25T19:44:21.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.160+0000 7efefbfff640 1 -- 172.21.15.44:0/1346848785 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7eff0804f080 con 0x7eff1413f710 2024-03-25T19:44:21.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.160+0000 7efefbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:21.161 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.160+0000 7efefbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:21.402 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.401+0000 7eff19bba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:21.402 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.401+0000 7eff19bba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:21.402 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.401+0000 7eff19bba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:44:21.402 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.401+0000 7eff19bba640 1 -- 172.21.15.44:0/1346848785 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff140627b0 con 0x7eff1413f710 2024-03-25T19:44:21.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.487+0000 7efefbfff640 1 -- 172.21.15.44:0/1346848785 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7eff08051180 con 0x7eff1413f710 2024-03-25T19:44:21.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.487+0000 7efefbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:21.488 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.487+0000 7efefbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:21.488 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:44:21.488 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:21.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.489+0000 7eff19bba640 10 librados: watch_flush enter 2024-03-25T19:44:21.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.489+0000 7eff19bba640 10 librados: watch_flush exit 2024-03-25T19:44:21.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.489+0000 7eff19bba640 20 client.4616.objecter shutdown clearing up homeless session... 2024-03-25T19:44:21.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.489+0000 7eff19bba640 10 client.4616.objecter successfully canceled tick 2024-03-25T19:44:21.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.489+0000 7eff19bba640 1 -- 172.21.15.44:0/1346848785 >> v1:172.21.15.111:6800/3883215793 conn(0x7eff0006df20 legacy=0x7eff000703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:21.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.489+0000 7eff19bba640 10 monclient: shutdown 2024-03-25T19:44:21.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.489+0000 7eff19bba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:21.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.489+0000 7eff19bba640 1 -- 172.21.15.44:0/1346848785 >> v1:172.21.15.44:6790/0 conn(0x7eff1413f710 legacy=0x7eff14135020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:21.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.489+0000 7eff19bba640 1 -- 172.21.15.44:0/1346848785 shutdown_connections 2024-03-25T19:44:21.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.489+0000 7eff19bba640 1 -- 172.21.15.44:0/1346848785 >> 172.21.15.44:0/1346848785 conn(0x7eff140b7ea0 msgr2=0x7eff14074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:21.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.490+0000 7eff19bba640 1 -- 172.21.15.44:0/1346848785 shutdown_connections 2024-03-25T19:44:21.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.490+0000 7eff19bba640 1 -- 172.21.15.44:0/1346848785 wait complete. 2024-03-25T19:44:21.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:21.490+0000 7eff19bba640 1 librados: shutdown 2024-03-25T19:44:21.503 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:28.310 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:28.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.427+0000 7fbd1e8e6640 10 monclient: get_monmap_and_config 2024-03-25T19:44:28.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.427+0000 7fbd1e8e6640 10 monclient: build_initial_monmap 2024-03-25T19:44:28.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.427+0000 7fbd1e8e6640 10 monclient: monmap: 2024-03-25T19:44:28.427 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:44:28.427 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:28.427 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:44:28.428294+0000 2024-03-25T19:44:28.427 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:44:28.428294+0000 2024-03-25T19:44:28.428 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:28.428 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:28.428 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:44:28.428 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:44:28.428 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:44:28.428 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:28.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.427+0000 7fbd1e8e6640 1 Processor -- start 2024-03-25T19:44:28.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.428+0000 7fbd1e8e6640 1 -- start start 2024-03-25T19:44:28.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.428+0000 7fbd1e8e6640 10 monclient: init 2024-03-25T19:44:28.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.428+0000 7fbd1e8e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:28.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.428+0000 7fbd1e8e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:28.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.428+0000 7fbd1e8e6640 10 monclient(hunting): picked mon.noname-c con 0x7fbd180b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T19:44:28.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.428+0000 7fbd1e8e6640 10 monclient(hunting): picked mon.noname-b con 0x7fbd180b1730 addr v1:172.21.15.44:6790/0 2024-03-25T19:44:28.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.428+0000 7fbd1e8e6640 10 monclient(hunting): picked mon.noname-a con 0x7fbd1814be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:44:28.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.428+0000 7fbd1e8e6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd180af940 con 0x7fbd1814be90 2024-03-25T19:44:28.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.429+0000 7fbd1e8e6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd180b0a70 con 0x7fbd180b4d60 2024-03-25T19:44:28.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.429+0000 7fbd1e8e6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd180b0d50 con 0x7fbd180b1730 2024-03-25T19:44:28.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.429+0000 7fbd1e8e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:28.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.429+0000 7fbd1e8e6640 10 monclient(hunting): authenticate will time out at 5932.599950s 2024-03-25T19:44:28.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.430+0000 7fbd17fff640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fbd180b4d60 0x7fbd1813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:46014/0 (socket says 172.21.15.44:46014) 2024-03-25T19:44:28.430 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.430+0000 7fbd17fff640 1 -- 172.21.15.44:0/4008944486 learned_addr learned my addr 172.21.15.44:0/4008944486 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:44:28.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.470+0000 7fbd16ffd640 1 -- 172.21.15.44:0/4008944486 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3037071825 0 0) 0x7fbd180af940 con 0x7fbd1814be90 2024-03-25T19:44:28.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.470+0000 7fbd16ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:28.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.470+0000 7fbd16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:28.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.470+0000 7fbd16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:28.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.470+0000 7fbd16ffd640 10 monclient(hunting): my global_id is 4641 2024-03-25T19:44:28.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.470+0000 7fbd16ffd640 1 -- 172.21.15.44:0/4008944486 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd00003520 con 0x7fbd1814be90 2024-03-25T19:44:28.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.555+0000 7fbd16ffd640 1 -- 172.21.15.44:0/4008944486 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3415719529 0 0) 0x7fbd180b0d50 con 0x7fbd180b1730 2024-03-25T19:44:28.555 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.555+0000 7fbd16ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:28.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.555+0000 7fbd16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:28.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.555+0000 7fbd16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:28.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.555+0000 7fbd16ffd640 10 monclient(hunting): my global_id is 4619 2024-03-25T19:44:28.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.555+0000 7fbd16ffd640 1 -- 172.21.15.44:0/4008944486 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd180af940 con 0x7fbd180b1730 2024-03-25T19:44:28.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.559+0000 7fbd16ffd640 1 -- 172.21.15.44:0/4008944486 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1899614502 0 0) 0x7fbd00003520 con 0x7fbd1814be90 2024-03-25T19:44:28.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.559+0000 7fbd16ffd640 1 -- 172.21.15.44:0/4008944486 >> v1:172.21.15.44:6790/0 conn(0x7fbd180b1730 legacy=0x7fbd180b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:28.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.559+0000 7fbd16ffd640 1 -- 172.21.15.44:0/4008944486 >> v1:172.21.15.111:6789/0 conn(0x7fbd180b4d60 legacy=0x7fbd1813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:28.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.559+0000 7fbd16ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:28.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.559+0000 7fbd16ffd640 1 monclient: found mon.noname-a 2024-03-25T19:44:28.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.559+0000 7fbd16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:28.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.559+0000 7fbd16ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:44:28.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.559+0000 7fbd16ffd640 1 -- 172.21.15.44:0/4008944486 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd180b1030 con 0x7fbd1814be90 2024-03-25T19:44:28.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.559+0000 7fbd16ffd640 10 monclient: _finish_auth 0 2024-03-25T19:44:28.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.559+0000 7fbd16ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:28.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.559+0000 7fbd16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:28.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.559+0000 7fbd16ffd640 1 -- 172.21.15.44:0/4008944486 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fbd10001a10 con 0x7fbd1814be90 2024-03-25T19:44:28.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.559+0000 7fbd16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:28.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.559+0000 7fbd16ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:28.560 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.559+0000 7fbd16ffd640 10 monclient: dump: 2024-03-25T19:44:28.560 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:28.560 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:28.560 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:28.561 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:28.561 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:28.561 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:28.561 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:28.561 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:28.561 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:28.561 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:28.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.559+0000 7fbd1e8e6640 5 monclient: authenticate success, global_id 4641 2024-03-25T19:44:28.561 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.559+0000 7fbd1e8e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:28.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.578+0000 7fbd16ffd640 1 -- 172.21.15.44:0/4008944486 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd10001b70 con 0x7fbd1814be90 2024-03-25T19:44:28.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.578+0000 7fbd16ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:28.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.578+0000 7fbd16ffd640 1 -- 172.21.15.44:0/4008944486 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fbd10001f80 con 0x7fbd1814be90 2024-03-25T19:44:28.578 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.578+0000 7fbd1e8e6640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:28.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.578+0000 7fbd1e8e6640 4 set_mon_vals no callback set 2024-03-25T19:44:28.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.578+0000 7fbd16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:28.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.578+0000 7fbd16ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:28.579 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.578+0000 7fbd16ffd640 10 monclient: dump: 2024-03-25T19:44:28.579 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:28.579 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:28.579 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:28.579 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:28.579 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:28.579 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:28.579 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:28.579 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:28.579 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:28.580 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:28.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.579+0000 7fbd1e8e6640 10 monclient: shutdown 2024-03-25T19:44:28.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.579+0000 7fbd1e8e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:28.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.579+0000 7fbd1e8e6640 1 -- 172.21.15.44:0/4008944486 >> v1:172.21.15.44:6789/0 conn(0x7fbd1814be90 legacy=0x7fbd1814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:28.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.579+0000 7fbd1e8e6640 1 -- 172.21.15.44:0/4008944486 shutdown_connections 2024-03-25T19:44:28.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.579+0000 7fbd1e8e6640 1 -- 172.21.15.44:0/4008944486 >> 172.21.15.44:0/4008944486 conn(0x7fbd180617a0 msgr2=0x7fbd18061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:28.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.579+0000 7fbd1e8e6640 1 -- 172.21.15.44:0/4008944486 shutdown_connections 2024-03-25T19:44:28.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.579+0000 7fbd1e8e6640 1 -- 172.21.15.44:0/4008944486 wait complete. 2024-03-25T19:44:28.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.580+0000 7fbd1e8e6640 10 monclient: build_initial_monmap 2024-03-25T19:44:28.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.580+0000 7fbd1e8e6640 10 monclient: monmap: 2024-03-25T19:44:28.583 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:44:28.583 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:28.583 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:44:28.583 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:44:28.583 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:28.583 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:44:28.583 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:44:28.583 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:44:28.583 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:44:28.583 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:28.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.580+0000 7fbd1e8e6640 1 librados: starting msgr at 2024-03-25T19:44:28.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.580+0000 7fbd1e8e6640 1 librados: starting objecter 2024-03-25T19:44:28.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.580+0000 7fbd1e8e6640 1 Processor -- start 2024-03-25T19:44:28.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.581+0000 7fbd1e8e6640 1 -- start start 2024-03-25T19:44:28.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.581+0000 7fbd1e8e6640 1 librados: setting wanted keys 2024-03-25T19:44:28.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.581+0000 7fbd1e8e6640 1 librados: calling monclient init 2024-03-25T19:44:28.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.581+0000 7fbd1e8e6640 10 monclient: init 2024-03-25T19:44:28.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.581+0000 7fbd1e8e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:28.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.581+0000 7fbd1e8e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:28.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.581+0000 7fbd1e8e6640 10 monclient(hunting): picked mon.noname-a con 0x7fbd180b1730 addr v1:172.21.15.44:6789/0 2024-03-25T19:44:28.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.581+0000 7fbd1e8e6640 10 monclient(hunting): picked mon.noname-c con 0x7fbd180b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T19:44:28.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.581+0000 7fbd1e8e6640 10 monclient(hunting): picked mon.noname-b con 0x7fbd1814be90 addr v1:172.21.15.111:6789/0 2024-03-25T19:44:28.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.581+0000 7fbd1e8e6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd18141ca0 con 0x7fbd180b1730 2024-03-25T19:44:28.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.582+0000 7fbd1e8e6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd181423a0 con 0x7fbd1814be90 2024-03-25T19:44:28.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.582+0000 7fbd1e8e6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd18140d00 con 0x7fbd180b4d60 2024-03-25T19:44:28.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.582+0000 7fbd1e8e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:28.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.582+0000 7fbd1e8e6640 10 monclient(hunting): authenticate will time out at 5932.752908s 2024-03-25T19:44:28.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.595+0000 7fbd177fe640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fbd180b4d60 0x7fbd18142b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:37308/0 (socket says 172.21.15.44:37308) 2024-03-25T19:44:28.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.595+0000 7fbd177fe640 1 -- 172.21.15.44:0/918172448 learned_addr learned my addr 172.21.15.44:0/918172448 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:44:28.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.597+0000 7fbd14ff9640 10 client.?.objecter ms_handle_connect 0x7fbd180b4d60 2024-03-25T19:44:28.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.597+0000 7fbd14ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:28.598 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.598+0000 7fbd14ff9640 10 client.?.objecter ms_handle_connect 0x7fbd1814be90 2024-03-25T19:44:28.598 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.598+0000 7fbd14ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:28.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.609+0000 7fbd14ff9640 10 client.?.objecter ms_handle_connect 0x7fbd180b1730 2024-03-25T19:44:28.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.609+0000 7fbd14ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:28.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.611+0000 7fbd14ff9640 1 -- 172.21.15.44:0/918172448 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885307654 0 0) 0x7fbd181423a0 con 0x7fbd1814be90 2024-03-25T19:44:28.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.611+0000 7fbd14ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:28.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.611+0000 7fbd14ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:28.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.611+0000 7fbd14ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:28.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.611+0000 7fbd14ff9640 10 monclient(hunting): my global_id is 4630 2024-03-25T19:44:28.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.611+0000 7fbd14ff9640 1 -- 172.21.15.44:0/918172448 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd04004eb0 con 0x7fbd1814be90 2024-03-25T19:44:28.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.619+0000 7fbd14ff9640 1 -- 172.21.15.44:0/918172448 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818060608 0 0) 0x7fbd18141ca0 con 0x7fbd180b1730 2024-03-25T19:44:28.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.619+0000 7fbd14ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:28.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.619+0000 7fbd14ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:28.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.619+0000 7fbd14ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:28.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.619+0000 7fbd14ff9640 10 monclient(hunting): my global_id is 4644 2024-03-25T19:44:28.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.619+0000 7fbd14ff9640 1 -- 172.21.15.44:0/918172448 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd181423a0 con 0x7fbd180b1730 2024-03-25T19:44:28.622 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.622+0000 7fbd14ff9640 1 -- 172.21.15.44:0/918172448 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315349565 0 0) 0x7fbd18140d00 con 0x7fbd180b4d60 2024-03-25T19:44:28.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.622+0000 7fbd14ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:28.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.622+0000 7fbd14ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:28.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.622+0000 7fbd14ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:28.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.622+0000 7fbd14ff9640 10 monclient(hunting): my global_id is 4622 2024-03-25T19:44:28.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.622+0000 7fbd14ff9640 1 -- 172.21.15.44:0/918172448 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd18141ca0 con 0x7fbd180b4d60 2024-03-25T19:44:28.631 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.631+0000 7fbd14ff9640 1 -- 172.21.15.44:0/918172448 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3957415690 0 0) 0x7fbd04004eb0 con 0x7fbd1814be90 2024-03-25T19:44:28.631 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.631+0000 7fbd14ff9640 1 -- 172.21.15.44:0/918172448 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd18140d00 con 0x7fbd1814be90 2024-03-25T19:44:28.631 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.631+0000 7fbd14ff9640 1 -- 172.21.15.44:0/918172448 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fbd10001ec0 con 0x7fbd1814be90 2024-03-25T19:44:28.631 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.631+0000 7fbd14ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:28.631 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.631+0000 7fbd14ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:28.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.631+0000 7fbd14ff9640 10 monclient(hunting): dump: 2024-03-25T19:44:28.632 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:28.632 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:28.632 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:28.632 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:28.632 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:28.632 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:28.632 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:28.632 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:28.632 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:28.632 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:28.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.631+0000 7fbd14ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:28.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.643+0000 7fbd14ff9640 1 -- 172.21.15.44:0/918172448 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1311837042 0 0) 0x7fbd181423a0 con 0x7fbd180b1730 2024-03-25T19:44:28.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.643+0000 7fbd14ff9640 1 -- 172.21.15.44:0/918172448 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd04004eb0 con 0x7fbd180b1730 2024-03-25T19:44:28.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.643+0000 7fbd14ff9640 1 -- 172.21.15.44:0/918172448 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fbd080018a0 con 0x7fbd180b1730 2024-03-25T19:44:28.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.643+0000 7fbd14ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:28.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.643+0000 7fbd14ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:28.643 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.643+0000 7fbd14ff9640 10 monclient(hunting): dump: 2024-03-25T19:44:28.644 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:28.644 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:28.644 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:28.644 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:28.644 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:28.644 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:28.644 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:28.644 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:28.644 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:28.644 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:28.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.644+0000 7fbd14ff9640 1 -- 172.21.15.44:0/918172448 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3317263 0 0) 0x7fbd18140d00 con 0x7fbd1814be90 2024-03-25T19:44:28.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.644+0000 7fbd14ff9640 1 -- 172.21.15.44:0/918172448 >> v1:172.21.15.44:6790/0 conn(0x7fbd180b4d60 legacy=0x7fbd18142b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:28.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.644+0000 7fbd14ff9640 1 -- 172.21.15.44:0/918172448 >> v1:172.21.15.44:6789/0 conn(0x7fbd180b1730 legacy=0x7fbd181405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:28.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.644+0000 7fbd14ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:44:28.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.644+0000 7fbd14ff9640 1 monclient: found mon.b 2024-03-25T19:44:28.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.644+0000 7fbd14ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:28.645 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.644+0000 7fbd14ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:44:28.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.644+0000 7fbd14ff9640 1 -- 172.21.15.44:0/918172448 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd18142640 con 0x7fbd1814be90 2024-03-25T19:44:28.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.644+0000 7fbd14ff9640 10 monclient: _finish_auth 0 2024-03-25T19:44:28.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.644+0000 7fbd14ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:44:28.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.644+0000 7fbd14ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:28.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.644+0000 7fbd1e8e6640 5 monclient: authenticate success, global_id 4630 2024-03-25T19:44:28.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.644+0000 7fbd1e8e6640 10 monclient: _renew_subs 2024-03-25T19:44:28.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.644+0000 7fbd1e8e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:44:28.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.644+0000 7fbd1e8e6640 1 -- 172.21.15.44:0/918172448 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd18143270 con 0x7fbd1814be90 2024-03-25T19:44:28.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.644+0000 7fbd1e8e6640 10 client.4630.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:28.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.644+0000 7fbd1e8e6640 10 monclient: _renew_subs 2024-03-25T19:44:28.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.644+0000 7fbd1e8e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:44:28.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.644+0000 7fbd1e8e6640 1 -- 172.21.15.44:0/918172448 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd181388c0 con 0x7fbd1814be90 2024-03-25T19:44:28.646 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.644+0000 7fbd1e8e6640 1 librados: init done 2024-03-25T19:44:28.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.645+0000 7fbd1e8e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:28.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.645+0000 7fbd1e8e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:28.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.645+0000 7fbd1e8e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:44:28.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.645+0000 7fbd1e8e6640 1 -- 172.21.15.44:0/918172448 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd180627b0 con 0x7fbd1814be90 2024-03-25T19:44:28.920 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.921+0000 7fbd14ff9640 1 -- 172.21.15.44:0/918172448 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd100020c0 con 0x7fbd1814be90 2024-03-25T19:44:28.920 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.921+0000 7fbd14ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:28.920 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.921+0000 7fbd14ff9640 1 -- 172.21.15.44:0/918172448 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fbd10003dc0 con 0x7fbd1814be90 2024-03-25T19:44:28.920 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.921+0000 7fbd14ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:28.920 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.921+0000 7fbd14ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:28.920 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.921+0000 7fbd14ff9640 10 monclient: dump: 2024-03-25T19:44:28.920 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:28.920 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:28.921 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:28.921 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:28.921 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:28.921 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:28.921 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:28.921 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:28.921 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:28.921 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:28.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.921+0000 7fbd167fc640 4 set_mon_vals no callback set 2024-03-25T19:44:28.922 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.921+0000 7fbd14ff9640 1 -- 172.21.15.44:0/918172448 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7fbd10019d90 con 0x7fbd1814be90 2024-03-25T19:44:28.922 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.922+0000 7fbd17fff640 10 monclient: get_auth_request con 0x7fbd0406df70 auth_method 0 2024-03-25T19:44:28.922 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.922+0000 7fbd14ff9640 10 client.4630.objecter ms_handle_connect 0x7fbd0406df70 2024-03-25T19:44:28.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.926+0000 7fbd14ff9640 1 -- 172.21.15.44:0/918172448 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7fbd10053920 con 0x7fbd1814be90 2024-03-25T19:44:28.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.926+0000 7fbd14ff9640 10 client.4630.objecter ms_dispatch 0x7fbd18000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:28.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.926+0000 7fbd14ff9640 3 client.4630.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:28.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.926+0000 7fbd14ff9640 3 client.4630.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:28.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.926+0000 7fbd14ff9640 20 client.4630.objecter dump_active .. 0 homeless 2024-03-25T19:44:28.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.926+0000 7fbd14ff9640 1 -- 172.21.15.44:0/918172448 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbd1004f430 con 0x7fbd1814be90 2024-03-25T19:44:28.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.926+0000 7fbd14ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:28.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:28.926+0000 7fbd14ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:29.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:29.186+0000 7fbd1e8e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:29.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:29.186+0000 7fbd1e8e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:29.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:29.186+0000 7fbd1e8e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:44:29.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:29.186+0000 7fbd1e8e6640 1 -- 172.21.15.44:0/918172448 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd180b8a90 con 0x7fbd1814be90 2024-03-25T19:44:29.204 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:29.205+0000 7fbd14ff9640 1 -- 172.21.15.44:0/918172448 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7fbd10051360 con 0x7fbd1814be90 2024-03-25T19:44:29.204 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:29.205+0000 7fbd14ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:29.204 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:29.205+0000 7fbd14ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:29.205 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:44:29.205 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:29.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:29.207+0000 7fbd1e8e6640 10 librados: watch_flush enter 2024-03-25T19:44:29.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:29.207+0000 7fbd1e8e6640 10 librados: watch_flush exit 2024-03-25T19:44:29.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:29.207+0000 7fbd1e8e6640 20 client.4630.objecter shutdown clearing up homeless session... 2024-03-25T19:44:29.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:29.207+0000 7fbd1e8e6640 10 client.4630.objecter successfully canceled tick 2024-03-25T19:44:29.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:29.207+0000 7fbd1e8e6640 1 -- 172.21.15.44:0/918172448 >> v1:172.21.15.111:6800/3883215793 conn(0x7fbd0406df70 legacy=0x7fbd04070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:29.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:29.207+0000 7fbd1e8e6640 10 monclient: shutdown 2024-03-25T19:44:29.207 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:29.207+0000 7fbd1e8e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:29.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:29.207+0000 7fbd1e8e6640 1 -- 172.21.15.44:0/918172448 >> v1:172.21.15.111:6789/0 conn(0x7fbd1814be90 legacy=0x7fbd181e22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:29.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:29.207+0000 7fbd1e8e6640 1 -- 172.21.15.44:0/918172448 shutdown_connections 2024-03-25T19:44:29.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:29.207+0000 7fbd1e8e6640 1 -- 172.21.15.44:0/918172448 >> 172.21.15.44:0/918172448 conn(0x7fbd180617a0 msgr2=0x7fbd180b2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:29.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:29.208+0000 7fbd1e8e6640 1 -- 172.21.15.44:0/918172448 shutdown_connections 2024-03-25T19:44:29.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:29.208+0000 7fbd1e8e6640 1 -- 172.21.15.44:0/918172448 wait complete. 2024-03-25T19:44:29.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:29.208+0000 7fbd1e8e6640 1 librados: shutdown 2024-03-25T19:44:29.218 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:36.025 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:36.138 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.138+0000 7f5a44315640 10 monclient: get_monmap_and_config 2024-03-25T19:44:36.138 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.138+0000 7f5a44315640 10 monclient: build_initial_monmap 2024-03-25T19:44:36.138 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.138+0000 7f5a44315640 10 monclient: monmap: 2024-03-25T19:44:36.138 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:44:36.138 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:36.138 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:44:36.139316+0000 2024-03-25T19:44:36.138 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:44:36.139316+0000 2024-03-25T19:44:36.139 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:36.139 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:36.139 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:44:36.139 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:44:36.139 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:44:36.139 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:36.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.138+0000 7f5a44315640 1 Processor -- start 2024-03-25T19:44:36.139 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.139+0000 7f5a44315640 1 -- start start 2024-03-25T19:44:36.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.139+0000 7f5a44315640 10 monclient: init 2024-03-25T19:44:36.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.140+0000 7f5a44315640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:36.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.140+0000 7f5a44315640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:36.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.140+0000 7f5a44315640 10 monclient(hunting): picked mon.noname-c con 0x7f5a3c0b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T19:44:36.140 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.140+0000 7f5a44315640 10 monclient(hunting): picked mon.noname-b con 0x7f5a3c0b1730 addr v1:172.21.15.44:6790/0 2024-03-25T19:44:36.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.140+0000 7f5a44315640 10 monclient(hunting): picked mon.noname-a con 0x7f5a3c14be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:44:36.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.140+0000 7f5a44315640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a3c0af940 con 0x7f5a3c14be90 2024-03-25T19:44:36.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.140+0000 7f5a44315640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a3c0b0a70 con 0x7f5a3c0b4d60 2024-03-25T19:44:36.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.140+0000 7f5a44315640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a3c0b0d50 con 0x7f5a3c0b1730 2024-03-25T19:44:36.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.140+0000 7f5a44315640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:36.141 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.140+0000 7f5a44315640 10 monclient(hunting): authenticate will time out at 5940.311173s 2024-03-25T19:44:36.158 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.158+0000 7f5a428b2640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f5a3c14be90 0x7f5a3c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:45152/0 (socket says 172.21.15.44:45152) 2024-03-25T19:44:36.158 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.158+0000 7f5a428b2640 1 -- 172.21.15.44:0/2799787859 learned_addr learned my addr 172.21.15.44:0/2799787859 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:44:36.169 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.169+0000 7f5a410af640 1 -- 172.21.15.44:0/2799787859 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644768059 0 0) 0x7f5a3c0af940 con 0x7f5a3c14be90 2024-03-25T19:44:36.169 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.169+0000 7f5a410af640 10 monclient(hunting): _negotiate 2024-03-25T19:44:36.169 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.169+0000 7f5a410af640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:36.169 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.169+0000 7f5a410af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:36.169 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.169+0000 7f5a410af640 10 monclient(hunting): my global_id is 4647 2024-03-25T19:44:36.169 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.169+0000 7f5a410af640 1 -- 172.21.15.44:0/2799787859 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a28003520 con 0x7f5a3c14be90 2024-03-25T19:44:36.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.172+0000 7f5a410af640 1 -- 172.21.15.44:0/2799787859 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3807530883 0 0) 0x7f5a3c0b0d50 con 0x7f5a3c0b1730 2024-03-25T19:44:36.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.172+0000 7f5a410af640 10 monclient(hunting): _negotiate 2024-03-25T19:44:36.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.172+0000 7f5a410af640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:36.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.172+0000 7f5a410af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:36.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.172+0000 7f5a410af640 10 monclient(hunting): my global_id is 4625 2024-03-25T19:44:36.173 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.172+0000 7f5a410af640 1 -- 172.21.15.44:0/2799787859 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a3c0af940 con 0x7f5a3c0b1730 2024-03-25T19:44:36.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.176+0000 7f5a410af640 1 -- 172.21.15.44:0/2799787859 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992950074 0 0) 0x7f5a3c0b0a70 con 0x7f5a3c0b4d60 2024-03-25T19:44:36.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.176+0000 7f5a410af640 10 monclient(hunting): _negotiate 2024-03-25T19:44:36.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.176+0000 7f5a410af640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:36.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.176+0000 7f5a410af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:36.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.176+0000 7f5a410af640 10 monclient(hunting): my global_id is 4633 2024-03-25T19:44:36.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.176+0000 7f5a410af640 1 -- 172.21.15.44:0/2799787859 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a3c0b0d50 con 0x7f5a3c0b4d60 2024-03-25T19:44:36.180 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.179+0000 7f5a410af640 1 -- 172.21.15.44:0/2799787859 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1946339597 0 0) 0x7f5a28003520 con 0x7f5a3c14be90 2024-03-25T19:44:36.180 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.179+0000 7f5a410af640 1 -- 172.21.15.44:0/2799787859 >> v1:172.21.15.44:6790/0 conn(0x7f5a3c0b1730 legacy=0x7f5a3c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:36.180 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.179+0000 7f5a410af640 1 -- 172.21.15.44:0/2799787859 >> v1:172.21.15.111:6789/0 conn(0x7f5a3c0b4d60 legacy=0x7f5a3c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:36.180 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.179+0000 7f5a410af640 10 monclient: _finish_hunting 0 2024-03-25T19:44:36.180 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.179+0000 7f5a410af640 1 monclient: found mon.noname-a 2024-03-25T19:44:36.180 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.179+0000 7f5a410af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:36.180 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.179+0000 7f5a410af640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:44:36.180 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.179+0000 7f5a410af640 1 -- 172.21.15.44:0/2799787859 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3c0b1030 con 0x7f5a3c14be90 2024-03-25T19:44:36.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.179+0000 7f5a410af640 10 monclient: _finish_auth 0 2024-03-25T19:44:36.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.179+0000 7f5a410af640 10 monclient: _check_auth_tickets 2024-03-25T19:44:36.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.179+0000 7f5a410af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:36.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.179+0000 7f5a410af640 1 -- 172.21.15.44:0/2799787859 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f5a34001690 con 0x7f5a3c14be90 2024-03-25T19:44:36.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.179+0000 7f5a410af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:36.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.179+0000 7f5a410af640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:36.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.179+0000 7f5a410af640 10 monclient: dump: 2024-03-25T19:44:36.181 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:36.181 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:36.181 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:36.181 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:36.181 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:36.181 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:36.181 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:36.181 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:36.182 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:36.182 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:36.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.179+0000 7f5a44315640 5 monclient: authenticate success, global_id 4647 2024-03-25T19:44:36.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.179+0000 7f5a44315640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:36.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.183+0000 7f5a410af640 1 -- 172.21.15.44:0/2799787859 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a34001870 con 0x7f5a3c14be90 2024-03-25T19:44:36.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.183+0000 7f5a410af640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:36.183 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.183+0000 7f5a410af640 1 -- 172.21.15.44:0/2799787859 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f5a34001c80 con 0x7f5a3c14be90 2024-03-25T19:44:36.184 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.183+0000 7f5a410af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:36.184 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.183+0000 7f5a410af640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:36.184 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.183+0000 7f5a410af640 10 monclient: dump: 2024-03-25T19:44:36.184 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:36.184 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:36.184 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:36.184 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:36.184 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:36.184 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:36.184 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:36.184 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:36.184 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:36.184 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:36.184 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.183+0000 7f5a44315640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:36.185 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.183+0000 7f5a44315640 4 set_mon_vals no callback set 2024-03-25T19:44:36.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.184+0000 7f5a44315640 10 monclient: shutdown 2024-03-25T19:44:36.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.184+0000 7f5a44315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:36.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.184+0000 7f5a44315640 1 -- 172.21.15.44:0/2799787859 >> v1:172.21.15.44:6789/0 conn(0x7f5a3c14be90 legacy=0x7f5a3c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:36.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.184+0000 7f5a44315640 1 -- 172.21.15.44:0/2799787859 shutdown_connections 2024-03-25T19:44:36.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.184+0000 7f5a44315640 1 -- 172.21.15.44:0/2799787859 >> 172.21.15.44:0/2799787859 conn(0x7f5a3c0617a0 msgr2=0x7f5a3c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:36.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.184+0000 7f5a44315640 1 -- 172.21.15.44:0/2799787859 shutdown_connections 2024-03-25T19:44:36.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.184+0000 7f5a44315640 1 -- 172.21.15.44:0/2799787859 wait complete. 2024-03-25T19:44:36.186 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.185+0000 7f5a44315640 10 monclient: build_initial_monmap 2024-03-25T19:44:36.187 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.185+0000 7f5a44315640 10 monclient: monmap: 2024-03-25T19:44:36.187 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:44:36.187 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:36.187 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:44:36.187 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:44:36.187 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:36.187 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:44:36.187 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:44:36.187 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:44:36.187 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:44:36.187 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:36.187 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.186+0000 7f5a44315640 1 librados: starting msgr at 2024-03-25T19:44:36.187 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.186+0000 7f5a44315640 1 librados: starting objecter 2024-03-25T19:44:36.187 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.186+0000 7f5a44315640 1 Processor -- start 2024-03-25T19:44:36.188 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.186+0000 7f5a44315640 1 -- start start 2024-03-25T19:44:36.188 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.186+0000 7f5a44315640 1 librados: setting wanted keys 2024-03-25T19:44:36.188 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.186+0000 7f5a44315640 1 librados: calling monclient init 2024-03-25T19:44:36.188 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.186+0000 7f5a44315640 10 monclient: init 2024-03-25T19:44:36.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.186+0000 7f5a44315640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:36.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.186+0000 7f5a44315640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:36.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.186+0000 7f5a44315640 10 monclient(hunting): picked mon.noname-b con 0x7f5a3c0b1730 addr v1:172.21.15.111:6789/0 2024-03-25T19:44:36.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.186+0000 7f5a44315640 10 monclient(hunting): picked mon.noname-c con 0x7f5a3c0b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T19:44:36.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.186+0000 7f5a44315640 10 monclient(hunting): picked mon.noname-a con 0x7f5a3c14be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:44:36.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.186+0000 7f5a44315640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a3c141950 con 0x7f5a3c14be90 2024-03-25T19:44:36.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.186+0000 7f5a44315640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a3c1d7940 con 0x7f5a3c0b1730 2024-03-25T19:44:36.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.186+0000 7f5a44315640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a3c1d8570 con 0x7f5a3c0b4d60 2024-03-25T19:44:36.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.186+0000 7f5a44315640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:36.189 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.186+0000 7f5a44315640 10 monclient(hunting): authenticate will time out at 5940.357774s 2024-03-25T19:44:36.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.202+0000 7f5a428b2640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f5a3c14be90 0x7f5a3c1e2fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:45178/0 (socket says 172.21.15.44:45178) 2024-03-25T19:44:36.202 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.202+0000 7f5a428b2640 1 -- 172.21.15.44:0/3792942776 learned_addr learned my addr 172.21.15.44:0/3792942776 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:44:36.204 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.203+0000 7f5a32ffd640 10 client.?.objecter ms_handle_connect 0x7f5a3c14be90 2024-03-25T19:44:36.204 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.203+0000 7f5a32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:36.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.208+0000 7f5a32ffd640 10 client.?.objecter ms_handle_connect 0x7f5a3c0b4d60 2024-03-25T19:44:36.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.208+0000 7f5a32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:36.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.228+0000 7f5a32ffd640 10 client.?.objecter ms_handle_connect 0x7f5a3c0b1730 2024-03-25T19:44:36.228 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.228+0000 7f5a32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:36.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.232+0000 7f5a32ffd640 1 -- 172.21.15.44:0/3792942776 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748558792 0 0) 0x7f5a3c141950 con 0x7f5a3c14be90 2024-03-25T19:44:36.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.232+0000 7f5a32ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:36.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.232+0000 7f5a32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:36.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.232+0000 7f5a32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:36.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.232+0000 7f5a32ffd640 10 monclient(hunting): my global_id is 4650 2024-03-25T19:44:36.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.232+0000 7f5a32ffd640 1 -- 172.21.15.44:0/3792942776 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a20004eb0 con 0x7f5a3c14be90 2024-03-25T19:44:36.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.235+0000 7f5a32ffd640 1 -- 172.21.15.44:0/3792942776 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959808354 0 0) 0x7f5a3c1d8570 con 0x7f5a3c0b4d60 2024-03-25T19:44:36.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.235+0000 7f5a32ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:36.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.235+0000 7f5a32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:36.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.235+0000 7f5a32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:36.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.235+0000 7f5a32ffd640 10 monclient(hunting): my global_id is 4628 2024-03-25T19:44:36.236 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.235+0000 7f5a32ffd640 1 -- 172.21.15.44:0/3792942776 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a3c141950 con 0x7f5a3c0b4d60 2024-03-25T19:44:36.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.251+0000 7f5a32ffd640 1 -- 172.21.15.44:0/3792942776 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2804542599 0 0) 0x7f5a3c1d7940 con 0x7f5a3c0b1730 2024-03-25T19:44:36.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.251+0000 7f5a32ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:36.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.251+0000 7f5a32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:36.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.251+0000 7f5a32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:36.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.251+0000 7f5a32ffd640 10 monclient(hunting): my global_id is 4636 2024-03-25T19:44:36.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.251+0000 7f5a32ffd640 1 -- 172.21.15.44:0/3792942776 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a3c1d8570 con 0x7f5a3c0b1730 2024-03-25T19:44:36.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.258+0000 7f5a32ffd640 1 -- 172.21.15.44:0/3792942776 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1077323793 0 0) 0x7f5a3c141950 con 0x7f5a3c0b4d60 2024-03-25T19:44:36.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.258+0000 7f5a32ffd640 1 -- 172.21.15.44:0/3792942776 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a3c1d7940 con 0x7f5a3c0b4d60 2024-03-25T19:44:36.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.258+0000 7f5a32ffd640 1 -- 172.21.15.44:0/3792942776 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f5a2c001930 con 0x7f5a3c0b4d60 2024-03-25T19:44:36.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.258+0000 7f5a32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:36.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.258+0000 7f5a32ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:36.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.258+0000 7f5a32ffd640 10 monclient(hunting): dump: 2024-03-25T19:44:36.259 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:36.259 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:36.259 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:36.259 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:36.259 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:36.259 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:36.259 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:36.259 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:36.259 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:36.259 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:36.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.258+0000 7f5a32ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:36.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.260+0000 7f5a32ffd640 1 -- 172.21.15.44:0/3792942776 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3611858549 0 0) 0x7f5a20004eb0 con 0x7f5a3c14be90 2024-03-25T19:44:36.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.260+0000 7f5a32ffd640 1 -- 172.21.15.44:0/3792942776 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a3c141950 con 0x7f5a3c14be90 2024-03-25T19:44:36.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.260+0000 7f5a32ffd640 1 -- 172.21.15.44:0/3792942776 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f5a34001bc0 con 0x7f5a3c14be90 2024-03-25T19:44:36.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.260+0000 7f5a32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:36.260 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.260+0000 7f5a32ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:36.261 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.260+0000 7f5a32ffd640 10 monclient(hunting): dump: 2024-03-25T19:44:36.261 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:36.261 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:36.261 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:36.261 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:36.261 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:36.261 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:36.261 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:36.261 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:36.261 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:36.261 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:36.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.269+0000 7f5a32ffd640 1 -- 172.21.15.44:0/3792942776 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3947459422 0 0) 0x7f5a3c1d7940 con 0x7f5a3c0b4d60 2024-03-25T19:44:36.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.269+0000 7f5a32ffd640 1 -- 172.21.15.44:0/3792942776 >> v1:172.21.15.111:6789/0 conn(0x7f5a3c0b1730 legacy=0x7f5a3c1d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:36.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.269+0000 7f5a32ffd640 1 -- 172.21.15.44:0/3792942776 >> v1:172.21.15.44:6789/0 conn(0x7f5a3c14be90 legacy=0x7f5a3c1e2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:36.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.269+0000 7f5a32ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:36.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.269+0000 7f5a32ffd640 1 monclient: found mon.c 2024-03-25T19:44:36.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.269+0000 7f5a32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:36.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.269+0000 7f5a32ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:44:36.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.269+0000 7f5a32ffd640 1 -- 172.21.15.44:0/3792942776 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3c138840 con 0x7f5a3c0b4d60 2024-03-25T19:44:36.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.269+0000 7f5a32ffd640 10 monclient: _finish_auth 0 2024-03-25T19:44:36.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.269+0000 7f5a32ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:36.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.269+0000 7f5a32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:36.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.269+0000 7f5a44315640 5 monclient: authenticate success, global_id 4628 2024-03-25T19:44:36.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.269+0000 7f5a44315640 10 monclient: _renew_subs 2024-03-25T19:44:36.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.269+0000 7f5a44315640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:44:36.270 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.269+0000 7f5a44315640 1 -- 172.21.15.44:0/3792942776 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a3c142f70 con 0x7f5a3c0b4d60 2024-03-25T19:44:36.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.269+0000 7f5a44315640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:36.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.269+0000 7f5a44315640 10 monclient: _renew_subs 2024-03-25T19:44:36.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.269+0000 7f5a44315640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:44:36.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.269+0000 7f5a44315640 1 -- 172.21.15.44:0/3792942776 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a3c1432c0 con 0x7f5a3c0b4d60 2024-03-25T19:44:36.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.269+0000 7f5a44315640 1 librados: init done 2024-03-25T19:44:36.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.270+0000 7f5a44315640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:36.271 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.270+0000 7f5a44315640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:36.272 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.270+0000 7f5a44315640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:44:36.272 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.270+0000 7f5a44315640 1 -- 172.21.15.44:0/3792942776 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a3c1d7ad0 con 0x7f5a3c0b4d60 2024-03-25T19:44:36.581 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.581+0000 7f5a32ffd640 1 -- 172.21.15.44:0/3792942776 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a2c001d40 con 0x7f5a3c0b4d60 2024-03-25T19:44:36.581 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.581+0000 7f5a32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:36.581 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.581+0000 7f5a32ffd640 1 -- 172.21.15.44:0/3792942776 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f5a2c0037b0 con 0x7f5a3c0b4d60 2024-03-25T19:44:36.581 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.581+0000 7f5a32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:36.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.581+0000 7f5a32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:36.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.581+0000 7f5a32ffd640 10 monclient: dump: 2024-03-25T19:44:36.582 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:36.582 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:36.582 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:36.582 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:36.582 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:36.582 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:36.582 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:36.582 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:36.582 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:36.582 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:36.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.581+0000 7f5a408ae640 4 set_mon_vals no callback set 2024-03-25T19:44:36.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.582+0000 7f5a32ffd640 1 -- 172.21.15.44:0/3792942776 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f5a2c019800 con 0x7f5a3c0b4d60 2024-03-25T19:44:36.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.582+0000 7f5a420b1640 10 monclient: get_auth_request con 0x7f5a2006df20 auth_method 0 2024-03-25T19:44:36.583 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.583+0000 7f5a32ffd640 10 client.4628.objecter ms_handle_connect 0x7f5a2006df20 2024-03-25T19:44:36.587 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.587+0000 7f5a32ffd640 1 -- 172.21.15.44:0/3792942776 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f5a2c053570 con 0x7f5a3c0b4d60 2024-03-25T19:44:36.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.587+0000 7f5a32ffd640 10 client.4628.objecter ms_dispatch 0x7f5a3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:36.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.587+0000 7f5a32ffd640 3 client.4628.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:36.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.587+0000 7f5a32ffd640 3 client.4628.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:36.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.587+0000 7f5a32ffd640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-25T19:44:36.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.587+0000 7f5a32ffd640 1 -- 172.21.15.44:0/3792942776 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a2c04eeb0 con 0x7f5a3c0b4d60 2024-03-25T19:44:36.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.587+0000 7f5a32ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:36.588 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.587+0000 7f5a32ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:36.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.805+0000 7f5a44315640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:36.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.805+0000 7f5a44315640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:36.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.805+0000 7f5a44315640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:44:36.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.805+0000 7f5a44315640 1 -- 172.21.15.44:0/3792942776 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a3c1389d0 con 0x7f5a3c0b4d60 2024-03-25T19:44:36.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.917+0000 7f5a32ffd640 1 -- 172.21.15.44:0/3792942776 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f5a2c050fb0 con 0x7f5a3c0b4d60 2024-03-25T19:44:36.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.917+0000 7f5a32ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:36.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.917+0000 7f5a32ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:36.918 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:44:36.918 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:36.920 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.920+0000 7f5a44315640 10 librados: watch_flush enter 2024-03-25T19:44:36.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.920+0000 7f5a44315640 10 librados: watch_flush exit 2024-03-25T19:44:36.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.920+0000 7f5a44315640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-25T19:44:36.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.920+0000 7f5a44315640 10 client.4628.objecter successfully canceled tick 2024-03-25T19:44:36.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.920+0000 7f5a44315640 1 -- 172.21.15.44:0/3792942776 >> v1:172.21.15.111:6800/3883215793 conn(0x7f5a2006df20 legacy=0x7f5a200703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:36.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.920+0000 7f5a44315640 10 monclient: shutdown 2024-03-25T19:44:36.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.920+0000 7f5a44315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:36.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.920+0000 7f5a44315640 1 -- 172.21.15.44:0/3792942776 >> v1:172.21.15.44:6790/0 conn(0x7f5a3c0b4d60 legacy=0x7f5a3c142860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:36.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.920+0000 7f5a44315640 1 -- 172.21.15.44:0/3792942776 shutdown_connections 2024-03-25T19:44:36.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.920+0000 7f5a44315640 1 -- 172.21.15.44:0/3792942776 >> 172.21.15.44:0/3792942776 conn(0x7f5a3c0617a0 msgr2=0x7f5a3c0b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:36.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.920+0000 7f5a44315640 1 -- 172.21.15.44:0/3792942776 shutdown_connections 2024-03-25T19:44:36.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.920+0000 7f5a44315640 1 -- 172.21.15.44:0/3792942776 wait complete. 2024-03-25T19:44:36.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:36.920+0000 7f5a44315640 1 librados: shutdown 2024-03-25T19:44:36.932 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:43.739 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:43.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.851+0000 7f2cc9257640 10 monclient: get_monmap_and_config 2024-03-25T19:44:43.851 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.851+0000 7f2cc9257640 10 monclient: build_initial_monmap 2024-03-25T19:44:43.852 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.851+0000 7f2cc9257640 10 monclient: monmap: 2024-03-25T19:44:43.852 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:44:43.852 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:43.852 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:44:43.852793+0000 2024-03-25T19:44:43.852 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:44:43.852793+0000 2024-03-25T19:44:43.852 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:43.852 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:43.852 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:44:43.852 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:44:43.852 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:44:43.852 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:43.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.852+0000 7f2cc9257640 1 Processor -- start 2024-03-25T19:44:43.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.852+0000 7f2cc9257640 1 -- start start 2024-03-25T19:44:43.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.852+0000 7f2cc9257640 10 monclient: init 2024-03-25T19:44:43.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.853+0000 7f2cc9257640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:43.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.853+0000 7f2cc9257640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:43.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.853+0000 7f2cc9257640 10 monclient(hunting): picked mon.noname-c con 0x7f2cc40a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:44:43.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.853+0000 7f2cc9257640 10 monclient(hunting): picked mon.noname-a con 0x7f2cc413f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:44:43.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.853+0000 7f2cc9257640 10 monclient(hunting): picked mon.noname-b con 0x7f2cc41430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:44:43.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.853+0000 7f2cc9257640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc4142520 con 0x7f2cc413f710 2024-03-25T19:44:43.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.853+0000 7f2cc9257640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc40b28a0 con 0x7f2cc40a98f0 2024-03-25T19:44:43.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.853+0000 7f2cc9257640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc40b2a30 con 0x7f2cc41430c0 2024-03-25T19:44:43.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.853+0000 7f2cc9257640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:43.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.853+0000 7f2cc9257640 10 monclient(hunting): authenticate will time out at 5948.024146s 2024-03-25T19:44:43.861 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.861+0000 7f2cc359e640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f2cc41430c0 0x7f2cc414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:51748/0 (socket says 172.21.15.44:51748) 2024-03-25T19:44:43.861 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.861+0000 7f2cc359e640 1 -- 172.21.15.44:0/1972448467 learned_addr learned my addr 172.21.15.44:0/1972448467 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:44:43.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.872+0000 7f2cc259c640 1 -- 172.21.15.44:0/1972448467 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712765232 0 0) 0x7f2cc40b2a30 con 0x7f2cc41430c0 2024-03-25T19:44:43.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.872+0000 7f2cc259c640 10 monclient(hunting): _negotiate 2024-03-25T19:44:43.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.872+0000 7f2cc259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:43.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.872+0000 7f2cc259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:43.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.872+0000 7f2cc259c640 10 monclient(hunting): my global_id is 4631 2024-03-25T19:44:43.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.872+0000 7f2cc259c640 1 -- 172.21.15.44:0/1972448467 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cb0003520 con 0x7f2cc41430c0 2024-03-25T19:44:43.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.889+0000 7f2cc259c640 1 -- 172.21.15.44:0/1972448467 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851938010 0 0) 0x7f2cc40b28a0 con 0x7f2cc40a98f0 2024-03-25T19:44:43.889 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.889+0000 7f2cc259c640 10 monclient(hunting): _negotiate 2024-03-25T19:44:43.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.889+0000 7f2cc259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:43.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.889+0000 7f2cc259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:43.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.889+0000 7f2cc259c640 10 monclient(hunting): my global_id is 4639 2024-03-25T19:44:43.890 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.889+0000 7f2cc259c640 1 -- 172.21.15.44:0/1972448467 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cc40b2a30 con 0x7f2cc40a98f0 2024-03-25T19:44:43.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.891+0000 7f2cc259c640 1 -- 172.21.15.44:0/1972448467 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759614533 0 0) 0x7f2cc4142520 con 0x7f2cc413f710 2024-03-25T19:44:43.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.891+0000 7f2cc259c640 10 monclient(hunting): _negotiate 2024-03-25T19:44:43.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.891+0000 7f2cc259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:43.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.891+0000 7f2cc259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:43.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.891+0000 7f2cc259c640 10 monclient(hunting): my global_id is 4653 2024-03-25T19:44:43.891 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.891+0000 7f2cc259c640 1 -- 172.21.15.44:0/1972448467 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cc40b28a0 con 0x7f2cc413f710 2024-03-25T19:44:43.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.893+0000 7f2cc259c640 1 -- 172.21.15.44:0/1972448467 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1584495279 0 0) 0x7f2cb0003520 con 0x7f2cc41430c0 2024-03-25T19:44:43.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.893+0000 7f2cc259c640 1 -- 172.21.15.44:0/1972448467 >> v1:172.21.15.111:6789/0 conn(0x7f2cc40a98f0 legacy=0x7f2cc40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:43.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.893+0000 7f2cc259c640 1 -- 172.21.15.44:0/1972448467 >> v1:172.21.15.44:6789/0 conn(0x7f2cc413f710 legacy=0x7f2cc41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:43.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.893+0000 7f2cc259c640 10 monclient: _finish_hunting 0 2024-03-25T19:44:43.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.893+0000 7f2cc259c640 1 monclient: found mon.noname-b 2024-03-25T19:44:43.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.893+0000 7f2cc259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:43.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.893+0000 7f2cc259c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:44:43.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.893+0000 7f2cc259c640 1 -- 172.21.15.44:0/1972448467 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc40b44c0 con 0x7f2cc41430c0 2024-03-25T19:44:43.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.893+0000 7f2cc259c640 10 monclient: _finish_auth 0 2024-03-25T19:44:43.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.893+0000 7f2cc259c640 10 monclient: _check_auth_tickets 2024-03-25T19:44:43.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.893+0000 7f2cc259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:43.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.893+0000 7f2cc259c640 1 -- 172.21.15.44:0/1972448467 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f2cbc001a10 con 0x7f2cc41430c0 2024-03-25T19:44:43.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.893+0000 7f2cc259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:43.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.893+0000 7f2cc259c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:43.894 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.893+0000 7f2cc259c640 10 monclient: dump: 2024-03-25T19:44:43.895 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:43.895 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:43.895 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:43.895 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:43.895 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:43.895 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:43.895 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:43.895 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:43.895 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:43.895 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:43.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.893+0000 7f2cc9257640 5 monclient: authenticate success, global_id 4631 2024-03-25T19:44:43.895 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.893+0000 7f2cc9257640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:43.905 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.905+0000 7f2cc259c640 1 -- 172.21.15.44:0/1972448467 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2cbc001bf0 con 0x7f2cc41430c0 2024-03-25T19:44:43.905 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.905+0000 7f2cc259c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:43.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.905+0000 7f2cc259c640 1 -- 172.21.15.44:0/1972448467 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f2cbc002000 con 0x7f2cc41430c0 2024-03-25T19:44:43.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.905+0000 7f2cc9257640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:43.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.905+0000 7f2cc9257640 4 set_mon_vals no callback set 2024-03-25T19:44:43.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.905+0000 7f2cc259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:43.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.905+0000 7f2cc259c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:43.906 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.905+0000 7f2cc259c640 10 monclient: dump: 2024-03-25T19:44:43.906 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:43.906 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:43.906 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:43.906 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:43.906 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:43.906 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:43.906 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:43.907 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:43.907 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:43.907 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:43.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.906+0000 7f2cc9257640 10 monclient: shutdown 2024-03-25T19:44:43.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.906+0000 7f2cc9257640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:43.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.906+0000 7f2cc9257640 1 -- 172.21.15.44:0/1972448467 >> v1:172.21.15.44:6790/0 conn(0x7f2cc41430c0 legacy=0x7f2cc414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:43.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.906+0000 7f2cc9257640 1 -- 172.21.15.44:0/1972448467 shutdown_connections 2024-03-25T19:44:43.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.906+0000 7f2cc9257640 1 -- 172.21.15.44:0/1972448467 >> 172.21.15.44:0/1972448467 conn(0x7f2cc40b7ea0 msgr2=0x7f2cc40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:43.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.906+0000 7f2cc9257640 1 -- 172.21.15.44:0/1972448467 shutdown_connections 2024-03-25T19:44:43.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.906+0000 7f2cc9257640 1 -- 172.21.15.44:0/1972448467 wait complete. 2024-03-25T19:44:43.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.907+0000 7f2cc9257640 10 monclient: build_initial_monmap 2024-03-25T19:44:43.908 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.907+0000 7f2cc9257640 10 monclient: monmap: 2024-03-25T19:44:43.909 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:44:43.909 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:43.909 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:44:43.909 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:44:43.909 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:43.909 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:44:43.909 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:44:43.909 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:44:43.909 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:44:43.909 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:43.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.907+0000 7f2cc9257640 1 librados: starting msgr at 2024-03-25T19:44:43.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.907+0000 7f2cc9257640 1 librados: starting objecter 2024-03-25T19:44:43.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.907+0000 7f2cc9257640 1 Processor -- start 2024-03-25T19:44:43.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.908+0000 7f2cc9257640 1 -- start start 2024-03-25T19:44:43.909 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.908+0000 7f2cc9257640 1 librados: setting wanted keys 2024-03-25T19:44:43.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.908+0000 7f2cc9257640 1 librados: calling monclient init 2024-03-25T19:44:43.910 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.908+0000 7f2cc9257640 10 monclient: init 2024-03-25T19:44:43.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.908+0000 7f2cc9257640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:43.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.908+0000 7f2cc9257640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:43.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.908+0000 7f2cc9257640 10 monclient(hunting): picked mon.noname-b con 0x7f2cc40a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:44:43.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.909+0000 7f2cc9257640 10 monclient(hunting): picked mon.noname-c con 0x7f2cc413f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:44:43.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.909+0000 7f2cc9257640 10 monclient(hunting): picked mon.noname-a con 0x7f2cc41430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:44:43.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.909+0000 7f2cc9257640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc4138b40 con 0x7f2cc41430c0 2024-03-25T19:44:43.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.909+0000 7f2cc9257640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc4138e20 con 0x7f2cc40a98f0 2024-03-25T19:44:43.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.909+0000 7f2cc9257640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc4139100 con 0x7f2cc413f710 2024-03-25T19:44:43.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.909+0000 7f2cc9257640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:43.911 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.909+0000 7f2cc9257640 10 monclient(hunting): authenticate will time out at 5948.080061s 2024-03-25T19:44:43.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.918+0000 7f2cc2d9d640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f2cc40a98f0 0x7f2cc4137380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:38822/0 (socket says 172.21.15.44:38822) 2024-03-25T19:44:43.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.918+0000 7f2cc2d9d640 1 -- 172.21.15.44:0/330322495 learned_addr learned my addr 172.21.15.44:0/330322495 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:44:43.919 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.919+0000 7f2cbb7fe640 10 client.?.objecter ms_handle_connect 0x7f2cc40a98f0 2024-03-25T19:44:43.919 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.919+0000 7f2cbb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:43.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.921+0000 7f2cbb7fe640 10 client.?.objecter ms_handle_connect 0x7f2cc413f710 2024-03-25T19:44:43.921 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.921+0000 7f2cbb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:43.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.925+0000 7f2cbb7fe640 1 -- 172.21.15.44:0/330322495 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862959335 0 0) 0x7f2cc4138e20 con 0x7f2cc40a98f0 2024-03-25T19:44:43.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.925+0000 7f2cbb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:43.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.925+0000 7f2cbb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:43.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.925+0000 7f2cbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:43.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.925+0000 7f2cbb7fe640 10 monclient(hunting): my global_id is 4642 2024-03-25T19:44:43.925 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.925+0000 7f2cbb7fe640 1 -- 172.21.15.44:0/330322495 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ca4004eb0 con 0x7f2cc40a98f0 2024-03-25T19:44:43.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.926+0000 7f2cbb7fe640 10 client.?.objecter ms_handle_connect 0x7f2cc41430c0 2024-03-25T19:44:43.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.926+0000 7f2cbb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:43.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.931+0000 7f2cbb7fe640 1 -- 172.21.15.44:0/330322495 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1769664579 0 0) 0x7f2ca4004eb0 con 0x7f2cc40a98f0 2024-03-25T19:44:43.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.931+0000 7f2cbb7fe640 1 -- 172.21.15.44:0/330322495 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cc4138e20 con 0x7f2cc40a98f0 2024-03-25T19:44:43.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.931+0000 7f2cbb7fe640 1 -- 172.21.15.44:0/330322495 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f2cb40019b0 con 0x7f2cc40a98f0 2024-03-25T19:44:43.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.931+0000 7f2cbb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:43.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.931+0000 7f2cbb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:43.932 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.931+0000 7f2cbb7fe640 10 monclient(hunting): dump: 2024-03-25T19:44:43.932 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:43.932 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:43.933 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:43.933 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:43.933 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:43.933 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:43.933 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:43.933 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:43.933 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:43.933 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:43.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.931+0000 7f2cbb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:43.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.934+0000 7f2cbb7fe640 1 -- 172.21.15.44:0/330322495 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2717089281 0 0) 0x7f2cc4139100 con 0x7f2cc413f710 2024-03-25T19:44:43.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.934+0000 7f2cbb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:43.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.934+0000 7f2cbb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:43.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.934+0000 7f2cbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:43.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.934+0000 7f2cbb7fe640 10 monclient(hunting): my global_id is 4634 2024-03-25T19:44:43.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.934+0000 7f2cbb7fe640 1 -- 172.21.15.44:0/330322495 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ca4004eb0 con 0x7f2cc413f710 2024-03-25T19:44:43.942 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.941+0000 7f2cbb7fe640 1 -- 172.21.15.44:0/330322495 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 970111963 0 0) 0x7f2cc4138e20 con 0x7f2cc40a98f0 2024-03-25T19:44:43.942 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.941+0000 7f2cbb7fe640 1 -- 172.21.15.44:0/330322495 >> v1:172.21.15.44:6790/0 conn(0x7f2cc413f710 legacy=0x7f2cc4137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:43.942 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.941+0000 7f2cbb7fe640 1 -- 172.21.15.44:0/330322495 >> v1:172.21.15.44:6789/0 conn(0x7f2cc41430c0 legacy=0x7f2cc41e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:43.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.942+0000 7f2cbb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:44:43.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.942+0000 7f2cbb7fe640 1 monclient: found mon.b 2024-03-25T19:44:43.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.942+0000 7f2cbb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:43.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.942+0000 7f2cbb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:44:43.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.942+0000 7f2cbb7fe640 1 -- 172.21.15.44:0/330322495 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc41397a0 con 0x7f2cc40a98f0 2024-03-25T19:44:43.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.942+0000 7f2cbb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:44:43.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.942+0000 7f2cbb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:44:43.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.942+0000 7f2cbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:43.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.942+0000 7f2cc9257640 5 monclient: authenticate success, global_id 4642 2024-03-25T19:44:43.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.942+0000 7f2cc9257640 10 monclient: _renew_subs 2024-03-25T19:44:43.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.942+0000 7f2cc9257640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:44:43.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.942+0000 7f2cc9257640 1 -- 172.21.15.44:0/330322495 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc4136aa0 con 0x7f2cc40a98f0 2024-03-25T19:44:43.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.942+0000 7f2cc9257640 10 client.4642.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:43.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.942+0000 7f2cc9257640 10 monclient: _renew_subs 2024-03-25T19:44:43.944 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.942+0000 7f2cc9257640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:44:43.944 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.942+0000 7f2cc9257640 1 -- 172.21.15.44:0/330322495 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc413c320 con 0x7f2cc40a98f0 2024-03-25T19:44:43.944 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.942+0000 7f2cc9257640 1 librados: init done 2024-03-25T19:44:43.944 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.943+0000 7f2cc9257640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:43.944 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.943+0000 7f2cc9257640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:43.944 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.943+0000 7f2cc9257640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:44:43.944 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:43.943+0000 7f2cc9257640 1 -- 172.21.15.44:0/330322495 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc41395b0 con 0x7f2cc40a98f0 2024-03-25T19:44:44.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.217+0000 7f2cbb7fe640 1 -- 172.21.15.44:0/330322495 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2cb4001c70 con 0x7f2cc40a98f0 2024-03-25T19:44:44.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.217+0000 7f2cbb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:44.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.218+0000 7f2cbb7fe640 1 -- 172.21.15.44:0/330322495 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f2cb4003a60 con 0x7f2cc40a98f0 2024-03-25T19:44:44.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.218+0000 7f2cbb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:44.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.218+0000 7f2cbb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:44.218 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.218+0000 7f2cbb7fe640 10 monclient: dump: 2024-03-25T19:44:44.219 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:44.219 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:44.219 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:44.219 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:44.219 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:44.219 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:44.219 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:44.219 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:44.219 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:44.219 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:44.219 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.217+0000 7f2cc1d9b640 4 set_mon_vals no callback set 2024-03-25T19:44:44.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.219+0000 7f2cbb7fe640 1 -- 172.21.15.44:0/330322495 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f2cb4003c40 con 0x7f2cc40a98f0 2024-03-25T19:44:44.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.219+0000 7f2cbbfff640 10 monclient: get_auth_request con 0x7f2ca406ddd0 auth_method 0 2024-03-25T19:44:44.220 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.220+0000 7f2cbb7fe640 10 client.4642.objecter ms_handle_connect 0x7f2ca406ddd0 2024-03-25T19:44:44.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.223+0000 7f2cbb7fe640 1 -- 172.21.15.44:0/330322495 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f2cb40547c0 con 0x7f2cc40a98f0 2024-03-25T19:44:44.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.223+0000 7f2cbb7fe640 10 client.4642.objecter ms_dispatch 0x7f2cc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:44.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.223+0000 7f2cbb7fe640 3 client.4642.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:44.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.223+0000 7f2cbb7fe640 3 client.4642.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:44.224 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.223+0000 7f2cbb7fe640 20 client.4642.objecter dump_active .. 0 homeless 2024-03-25T19:44:44.225 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.225+0000 7f2cbb7fe640 1 -- 172.21.15.44:0/330322495 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2cb40502d0 con 0x7f2cc40a98f0 2024-03-25T19:44:44.225 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.225+0000 7f2cbb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:44.225 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.225+0000 7f2cbb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:44.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.481+0000 7f2cc9257640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:44.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.481+0000 7f2cc9257640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:44.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.481+0000 7f2cc9257640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:44:44.482 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.481+0000 7f2cc9257640 1 -- 172.21.15.44:0/330322495 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2cc4139930 con 0x7f2cc40a98f0 2024-03-25T19:44:44.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.500+0000 7f2cbb7fe640 1 -- 172.21.15.44:0/330322495 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f2cb4052200 con 0x7f2cc40a98f0 2024-03-25T19:44:44.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.500+0000 7f2cbb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:44.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.500+0000 7f2cbb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:44.500 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:44:44.500 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:44.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.502+0000 7f2cc9257640 10 librados: watch_flush enter 2024-03-25T19:44:44.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.502+0000 7f2cc9257640 10 librados: watch_flush exit 2024-03-25T19:44:44.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.502+0000 7f2cc9257640 20 client.4642.objecter shutdown clearing up homeless session... 2024-03-25T19:44:44.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.502+0000 7f2cc9257640 10 client.4642.objecter successfully canceled tick 2024-03-25T19:44:44.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.502+0000 7f2cc9257640 1 -- 172.21.15.44:0/330322495 >> v1:172.21.15.111:6800/3883215793 conn(0x7f2ca406ddd0 legacy=0x7f2ca4070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:44.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.502+0000 7f2cc9257640 10 monclient: shutdown 2024-03-25T19:44:44.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.502+0000 7f2cc9257640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:44.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.502+0000 7f2cc9257640 1 -- 172.21.15.44:0/330322495 >> v1:172.21.15.111:6789/0 conn(0x7f2cc40a98f0 legacy=0x7f2cc4137380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:44.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.502+0000 7f2cc9257640 1 -- 172.21.15.44:0/330322495 shutdown_connections 2024-03-25T19:44:44.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.502+0000 7f2cc9257640 1 -- 172.21.15.44:0/330322495 >> 172.21.15.44:0/330322495 conn(0x7f2cc40b7ea0 msgr2=0x7f2cc414ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:44.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.502+0000 7f2cc9257640 1 -- 172.21.15.44:0/330322495 shutdown_connections 2024-03-25T19:44:44.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.502+0000 7f2cc9257640 1 -- 172.21.15.44:0/330322495 wait complete. 2024-03-25T19:44:44.504 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:44.502+0000 7f2cc9257640 1 librados: shutdown 2024-03-25T19:44:44.514 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:45.379 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:09:36.281 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:45.437 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:09:36.338 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:45.476 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:09:36.377 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:45.484 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:09:36.364 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:45.503 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:09:36.383 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:45.503 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:09:36.405 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:45.511 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:09:36.413 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:45.523 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:09:36.403 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:45.548 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:09:36.450 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:51.322 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:51.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.440+0000 7ff83b2ce640 10 monclient: get_monmap_and_config 2024-03-25T19:44:51.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.440+0000 7ff83b2ce640 10 monclient: build_initial_monmap 2024-03-25T19:44:51.440 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.440+0000 7ff83b2ce640 10 monclient: monmap: 2024-03-25T19:44:51.440 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:44:51.441 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:51.441 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:44:51.441444+0000 2024-03-25T19:44:51.441 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:44:51.441444+0000 2024-03-25T19:44:51.441 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:51.441 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:51.441 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:44:51.441 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:44:51.441 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:44:51.441 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:51.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.440+0000 7ff83b2ce640 1 Processor -- start 2024-03-25T19:44:51.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.441+0000 7ff83b2ce640 1 -- start start 2024-03-25T19:44:51.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.441+0000 7ff83b2ce640 10 monclient: init 2024-03-25T19:44:51.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.441+0000 7ff83b2ce640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:51.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.441+0000 7ff83b2ce640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:51.442 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.441+0000 7ff83b2ce640 10 monclient(hunting): picked mon.noname-a con 0x7ff8340a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:44:51.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.441+0000 7ff83b2ce640 10 monclient(hunting): picked mon.noname-b con 0x7ff83413f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:44:51.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.441+0000 7ff83b2ce640 10 monclient(hunting): picked mon.noname-c con 0x7ff8341430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:44:51.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.441+0000 7ff83b2ce640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff834142520 con 0x7ff8340a98f0 2024-03-25T19:44:51.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.441+0000 7ff83b2ce640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8340b28a0 con 0x7ff8341430c0 2024-03-25T19:44:51.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.441+0000 7ff83b2ce640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8340b2a30 con 0x7ff83413f710 2024-03-25T19:44:51.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.441+0000 7ff83b2ce640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:51.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.441+0000 7ff83b2ce640 10 monclient(hunting): authenticate will time out at 5955.612705s 2024-03-25T19:44:51.445 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.445+0000 7ff83986b640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff8341430c0 0x7ff83414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:46512/0 (socket says 172.21.15.44:46512) 2024-03-25T19:44:51.445 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.445+0000 7ff83986b640 1 -- 172.21.15.44:0/990045881 learned_addr learned my addr 172.21.15.44:0/990045881 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:44:51.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.461+0000 7ff82bfff640 1 -- 172.21.15.44:0/990045881 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 980508924 0 0) 0x7ff8340b28a0 con 0x7ff8341430c0 2024-03-25T19:44:51.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.461+0000 7ff82bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:51.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.461+0000 7ff82bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:51.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.461+0000 7ff82bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:51.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.461+0000 7ff82bfff640 10 monclient(hunting): my global_id is 4645 2024-03-25T19:44:51.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.461+0000 7ff82bfff640 1 -- 172.21.15.44:0/990045881 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff820003520 con 0x7ff8341430c0 2024-03-25T19:44:51.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.462+0000 7ff82bfff640 1 -- 172.21.15.44:0/990045881 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729156929 0 0) 0x7ff834142520 con 0x7ff8340a98f0 2024-03-25T19:44:51.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.462+0000 7ff82bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:51.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.462+0000 7ff82bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:51.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.462+0000 7ff82bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:51.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.462+0000 7ff82bfff640 10 monclient(hunting): my global_id is 4659 2024-03-25T19:44:51.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.462+0000 7ff82bfff640 1 -- 172.21.15.44:0/990045881 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8340b28a0 con 0x7ff8340a98f0 2024-03-25T19:44:51.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.466+0000 7ff82bfff640 1 -- 172.21.15.44:0/990045881 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1259057285 0 0) 0x7ff820003520 con 0x7ff8341430c0 2024-03-25T19:44:51.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.466+0000 7ff82bfff640 1 -- 172.21.15.44:0/990045881 >> v1:172.21.15.44:6790/0 conn(0x7ff83413f710 legacy=0x7ff8341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:51.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.466+0000 7ff82bfff640 1 -- 172.21.15.44:0/990045881 >> v1:172.21.15.44:6789/0 conn(0x7ff8340a98f0 legacy=0x7ff8340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:51.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.467+0000 7ff82bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:44:51.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.467+0000 7ff82bfff640 1 monclient: found mon.noname-c 2024-03-25T19:44:51.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.467+0000 7ff82bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:51.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.467+0000 7ff82bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:44:51.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.467+0000 7ff82bfff640 1 -- 172.21.15.44:0/990045881 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8340b44c0 con 0x7ff8341430c0 2024-03-25T19:44:51.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.467+0000 7ff82bfff640 10 monclient: _finish_auth 0 2024-03-25T19:44:51.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.467+0000 7ff82bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:44:51.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.467+0000 7ff82bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:51.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.467+0000 7ff82bfff640 1 -- 172.21.15.44:0/990045881 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff82c001a10 con 0x7ff8341430c0 2024-03-25T19:44:51.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.467+0000 7ff82bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:51.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.467+0000 7ff82bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:51.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.467+0000 7ff82bfff640 10 monclient: dump: 2024-03-25T19:44:51.468 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:51.468 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:51.468 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:51.468 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:51.468 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:51.468 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:51.468 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:51.468 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:51.468 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:51.468 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:51.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.467+0000 7ff83b2ce640 5 monclient: authenticate success, global_id 4645 2024-03-25T19:44:51.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.467+0000 7ff83b2ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:51.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.471+0000 7ff82bfff640 1 -- 172.21.15.44:0/990045881 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff82c001b70 con 0x7ff8341430c0 2024-03-25T19:44:51.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.471+0000 7ff82bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:51.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.471+0000 7ff82bfff640 1 -- 172.21.15.44:0/990045881 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff82c001f80 con 0x7ff8341430c0 2024-03-25T19:44:51.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.471+0000 7ff83b2ce640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:51.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.471+0000 7ff83b2ce640 4 set_mon_vals no callback set 2024-03-25T19:44:51.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.471+0000 7ff82bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:51.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.471+0000 7ff82bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:51.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.471+0000 7ff82bfff640 10 monclient: dump: 2024-03-25T19:44:51.472 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:51.472 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:51.472 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:51.472 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:51.472 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:51.472 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:51.472 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:51.472 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:51.472 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:51.472 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:51.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.472+0000 7ff83b2ce640 10 monclient: shutdown 2024-03-25T19:44:51.475 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.472+0000 7ff83b2ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:51.475 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.472+0000 7ff83b2ce640 1 -- 172.21.15.44:0/990045881 >> v1:172.21.15.111:6789/0 conn(0x7ff8341430c0 legacy=0x7ff83414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:51.475 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.472+0000 7ff83b2ce640 1 -- 172.21.15.44:0/990045881 shutdown_connections 2024-03-25T19:44:51.475 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.472+0000 7ff83b2ce640 1 -- 172.21.15.44:0/990045881 >> 172.21.15.44:0/990045881 conn(0x7ff8340b7ea0 msgr2=0x7ff8340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:51.475 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.472+0000 7ff83b2ce640 1 -- 172.21.15.44:0/990045881 shutdown_connections 2024-03-25T19:44:51.475 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.472+0000 7ff83b2ce640 1 -- 172.21.15.44:0/990045881 wait complete. 2024-03-25T19:44:51.475 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.473+0000 7ff83b2ce640 10 monclient: build_initial_monmap 2024-03-25T19:44:51.475 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.473+0000 7ff83b2ce640 10 monclient: monmap: 2024-03-25T19:44:51.475 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:44:51.475 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:51.475 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:44:51.475 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:44:51.475 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:51.476 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:44:51.476 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:44:51.476 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:44:51.476 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:44:51.476 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:51.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.473+0000 7ff83b2ce640 1 librados: starting msgr at 2024-03-25T19:44:51.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.473+0000 7ff83b2ce640 1 librados: starting objecter 2024-03-25T19:44:51.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.473+0000 7ff83b2ce640 1 Processor -- start 2024-03-25T19:44:51.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.473+0000 7ff83b2ce640 1 -- start start 2024-03-25T19:44:51.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.473+0000 7ff83b2ce640 1 librados: setting wanted keys 2024-03-25T19:44:51.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.473+0000 7ff83b2ce640 1 librados: calling monclient init 2024-03-25T19:44:51.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.473+0000 7ff83b2ce640 10 monclient: init 2024-03-25T19:44:51.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.474+0000 7ff83b2ce640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:51.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.474+0000 7ff83b2ce640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:51.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.474+0000 7ff83b2ce640 10 monclient(hunting): picked mon.noname-b con 0x7ff8340a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:44:51.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.474+0000 7ff83b2ce640 10 monclient(hunting): picked mon.noname-a con 0x7ff83413f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:44:51.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.474+0000 7ff83b2ce640 10 monclient(hunting): picked mon.noname-c con 0x7ff8341430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:44:51.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.474+0000 7ff83b2ce640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff83413c2b0 con 0x7ff83413f710 2024-03-25T19:44:51.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.474+0000 7ff83b2ce640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff834136f10 con 0x7ff8340a98f0 2024-03-25T19:44:51.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.474+0000 7ff83b2ce640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8341370a0 con 0x7ff8341430c0 2024-03-25T19:44:51.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.474+0000 7ff83b2ce640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:51.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.474+0000 7ff83b2ce640 10 monclient(hunting): authenticate will time out at 5955.645305s 2024-03-25T19:44:51.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.477+0000 7ff83906a640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff8340a98f0 0x7ff8341d7590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:46540/0 (socket says 172.21.15.44:46540) 2024-03-25T19:44:51.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.477+0000 7ff83906a640 1 -- 172.21.15.44:0/2159709608 learned_addr learned my addr 172.21.15.44:0/2159709608 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:44:51.478 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.478+0000 7ff829ffb640 10 client.?.objecter ms_handle_connect 0x7ff8340a98f0 2024-03-25T19:44:51.478 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.478+0000 7ff829ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:51.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.481+0000 7ff829ffb640 10 client.?.objecter ms_handle_connect 0x7ff83413f710 2024-03-25T19:44:51.481 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.481+0000 7ff829ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:51.483 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.483+0000 7ff829ffb640 1 -- 172.21.15.44:0/2159709608 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688682232 0 0) 0x7ff834136f10 con 0x7ff8340a98f0 2024-03-25T19:44:51.483 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.483+0000 7ff829ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:51.483 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.483+0000 7ff829ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:51.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.483+0000 7ff829ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:51.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.483+0000 7ff829ffb640 10 monclient(hunting): my global_id is 4648 2024-03-25T19:44:51.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.483+0000 7ff829ffb640 1 -- 172.21.15.44:0/2159709608 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff818004eb0 con 0x7ff8340a98f0 2024-03-25T19:44:51.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.490+0000 7ff829ffb640 1 -- 172.21.15.44:0/2159709608 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2065500207 0 0) 0x7ff83413c2b0 con 0x7ff83413f710 2024-03-25T19:44:51.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.490+0000 7ff829ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:51.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.490+0000 7ff829ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:51.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.490+0000 7ff829ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:51.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.490+0000 7ff829ffb640 10 monclient(hunting): my global_id is 4662 2024-03-25T19:44:51.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.490+0000 7ff829ffb640 1 -- 172.21.15.44:0/2159709608 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff834136f10 con 0x7ff83413f710 2024-03-25T19:44:51.492 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.492+0000 7ff829ffb640 10 client.?.objecter ms_handle_connect 0x7ff8341430c0 2024-03-25T19:44:51.492 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.492+0000 7ff829ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:51.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.493+0000 7ff829ffb640 1 -- 172.21.15.44:0/2159709608 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3004246441 0 0) 0x7ff818004eb0 con 0x7ff8340a98f0 2024-03-25T19:44:51.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.493+0000 7ff829ffb640 1 -- 172.21.15.44:0/2159709608 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff83413c2b0 con 0x7ff8340a98f0 2024-03-25T19:44:51.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.493+0000 7ff829ffb640 1 -- 172.21.15.44:0/2159709608 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff830001a10 con 0x7ff8340a98f0 2024-03-25T19:44:51.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.493+0000 7ff829ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:51.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.493+0000 7ff829ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:51.493 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.493+0000 7ff829ffb640 10 monclient(hunting): dump: 2024-03-25T19:44:51.493 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:51.493 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:51.493 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:51.493 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:51.493 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:51.493 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:51.494 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:51.494 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:51.494 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:51.494 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:51.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.493+0000 7ff829ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:51.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.499+0000 7ff829ffb640 1 -- 172.21.15.44:0/2159709608 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3807190007 0 0) 0x7ff83413c2b0 con 0x7ff8340a98f0 2024-03-25T19:44:51.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.500+0000 7ff829ffb640 1 -- 172.21.15.44:0/2159709608 >> v1:172.21.15.44:6790/0 conn(0x7ff8341430c0 legacy=0x7ff834133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:51.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.500+0000 7ff829ffb640 1 -- 172.21.15.44:0/2159709608 >> v1:172.21.15.44:6789/0 conn(0x7ff83413f710 legacy=0x7ff8341377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:51.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.500+0000 7ff829ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:44:51.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.500+0000 7ff829ffb640 1 monclient: found mon.b 2024-03-25T19:44:51.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.500+0000 7ff829ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:51.500 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.500+0000 7ff829ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:44:51.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.500+0000 7ff829ffb640 1 -- 172.21.15.44:0/2159709608 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff834137230 con 0x7ff8340a98f0 2024-03-25T19:44:51.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.500+0000 7ff829ffb640 10 monclient: _finish_auth 0 2024-03-25T19:44:51.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.500+0000 7ff829ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:44:51.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.500+0000 7ff829ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:51.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.500+0000 7ff83b2ce640 5 monclient: authenticate success, global_id 4648 2024-03-25T19:44:51.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.500+0000 7ff83b2ce640 10 monclient: _renew_subs 2024-03-25T19:44:51.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.500+0000 7ff83b2ce640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:44:51.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.500+0000 7ff83b2ce640 1 -- 172.21.15.44:0/2159709608 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff83413b620 con 0x7ff8340a98f0 2024-03-25T19:44:51.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.500+0000 7ff83b2ce640 10 client.4648.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:51.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.500+0000 7ff83b2ce640 10 monclient: _renew_subs 2024-03-25T19:44:51.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.500+0000 7ff83b2ce640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:44:51.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.500+0000 7ff83b2ce640 1 -- 172.21.15.44:0/2159709608 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff834133710 con 0x7ff8340a98f0 2024-03-25T19:44:51.501 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.500+0000 7ff83b2ce640 1 librados: init done 2024-03-25T19:44:51.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.501+0000 7ff83b2ce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:51.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.501+0000 7ff83b2ce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:51.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.501+0000 7ff83b2ce640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:44:51.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.501+0000 7ff83b2ce640 1 -- 172.21.15.44:0/2159709608 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8340627b0 con 0x7ff8340a98f0 2024-03-25T19:44:51.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.505+0000 7ff829ffb640 1 -- 172.21.15.44:0/2159709608 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff830001c50 con 0x7ff8340a98f0 2024-03-25T19:44:51.505 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.505+0000 7ff829ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:51.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.505+0000 7ff829ffb640 1 -- 172.21.15.44:0/2159709608 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff8300038c0 con 0x7ff8340a98f0 2024-03-25T19:44:51.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.505+0000 7ff829ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:51.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.505+0000 7ff829ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:51.506 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.505+0000 7ff829ffb640 10 monclient: dump: 2024-03-25T19:44:51.506 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:51.506 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:51.506 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:51.506 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:51.506 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:51.506 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:51.506 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:51.507 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:51.507 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:51.507 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:51.507 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.505+0000 7ff82b7fe640 4 set_mon_vals no callback set 2024-03-25T19:44:51.522 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.522+0000 7ff829ffb640 1 -- 172.21.15.44:0/2159709608 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7ff830003a20 con 0x7ff8340a98f0 2024-03-25T19:44:51.522 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.522+0000 7ff838869640 10 monclient: get_auth_request con 0x7ff81806dd10 auth_method 0 2024-03-25T19:44:51.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.523+0000 7ff829ffb640 10 client.4648.objecter ms_handle_connect 0x7ff81806dd10 2024-03-25T19:44:51.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.785+0000 7ff829ffb640 1 -- 172.21.15.44:0/2159709608 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7ff830054730 con 0x7ff8340a98f0 2024-03-25T19:44:51.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.785+0000 7ff829ffb640 10 client.4648.objecter ms_dispatch 0x7ff834000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:51.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.785+0000 7ff829ffb640 3 client.4648.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:51.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.785+0000 7ff829ffb640 3 client.4648.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:51.786 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.786+0000 7ff829ffb640 20 client.4648.objecter dump_active .. 0 homeless 2024-03-25T19:44:51.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.792+0000 7ff829ffb640 1 -- 172.21.15.44:0/2159709608 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff830050240 con 0x7ff8340a98f0 2024-03-25T19:44:51.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.792+0000 7ff829ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:51.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:51.792+0000 7ff829ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:52.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:52.038+0000 7ff83b2ce640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:52.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:52.038+0000 7ff83b2ce640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:52.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:52.038+0000 7ff83b2ce640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:44:52.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:52.038+0000 7ff83b2ce640 1 -- 172.21.15.44:0/2159709608 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8341373f0 con 0x7ff8340a98f0 2024-03-25T19:44:52.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:52.061+0000 7ff829ffb640 1 -- 172.21.15.44:0/2159709608 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7ff830052170 con 0x7ff8340a98f0 2024-03-25T19:44:52.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:52.061+0000 7ff829ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:52.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:52.061+0000 7ff829ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:52.062 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:44:52.062 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:52.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:52.064+0000 7ff83b2ce640 10 librados: watch_flush enter 2024-03-25T19:44:52.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:52.064+0000 7ff83b2ce640 10 librados: watch_flush exit 2024-03-25T19:44:52.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:52.064+0000 7ff83b2ce640 20 client.4648.objecter shutdown clearing up homeless session... 2024-03-25T19:44:52.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:52.064+0000 7ff83b2ce640 10 client.4648.objecter successfully canceled tick 2024-03-25T19:44:52.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:52.064+0000 7ff83b2ce640 1 -- 172.21.15.44:0/2159709608 >> v1:172.21.15.111:6800/3883215793 conn(0x7ff81806dd10 legacy=0x7ff8180701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:52.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:52.064+0000 7ff83b2ce640 10 monclient: shutdown 2024-03-25T19:44:52.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:52.064+0000 7ff83b2ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:52.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:52.064+0000 7ff83b2ce640 1 -- 172.21.15.44:0/2159709608 >> v1:172.21.15.111:6789/0 conn(0x7ff8340a98f0 legacy=0x7ff8341d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:52.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:52.064+0000 7ff83b2ce640 1 -- 172.21.15.44:0/2159709608 shutdown_connections 2024-03-25T19:44:52.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:52.064+0000 7ff83b2ce640 1 -- 172.21.15.44:0/2159709608 >> 172.21.15.44:0/2159709608 conn(0x7ff8340b7ea0 msgr2=0x7ff834074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:52.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:52.064+0000 7ff83b2ce640 1 -- 172.21.15.44:0/2159709608 shutdown_connections 2024-03-25T19:44:52.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:52.064+0000 7ff83b2ce640 1 -- 172.21.15.44:0/2159709608 wait complete. 2024-03-25T19:44:52.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:52.064+0000 7ff83b2ce640 1 librados: shutdown 2024-03-25T19:44:52.077 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:58.883 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:58.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:58.994+0000 7f2169f65640 10 monclient: get_monmap_and_config 2024-03-25T19:44:58.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:58.994+0000 7f2169f65640 10 monclient: build_initial_monmap 2024-03-25T19:44:58.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:58.995+0000 7f2169f65640 10 monclient: monmap: 2024-03-25T19:44:58.995 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:44:58.995 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:58.995 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:44:58.996133+0000 2024-03-25T19:44:58.995 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:44:58.996133+0000 2024-03-25T19:44:58.995 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:58.995 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:58.995 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:44:58.996 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:44:58.996 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:44:58.996 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:58.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:58.996+0000 7f2169f65640 1 Processor -- start 2024-03-25T19:44:58.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:58.996+0000 7f2169f65640 1 -- start start 2024-03-25T19:44:58.996 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:58.996+0000 7f2169f65640 10 monclient: init 2024-03-25T19:44:58.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:58.996+0000 7f2169f65640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:58.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:58.996+0000 7f2169f65640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:58.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:58.997+0000 7f2169f65640 10 monclient(hunting): picked mon.noname-c con 0x7f21640a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:44:58.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:58.997+0000 7f2169f65640 10 monclient(hunting): picked mon.noname-a con 0x7f216413f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:44:58.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:58.997+0000 7f2169f65640 10 monclient(hunting): picked mon.noname-b con 0x7f21641430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:44:58.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:58.997+0000 7f2169f65640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2164142520 con 0x7f216413f710 2024-03-25T19:44:58.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:58.997+0000 7f2169f65640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21640b28a0 con 0x7f21640a98f0 2024-03-25T19:44:58.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:58.997+0000 7f2169f65640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21640b2a30 con 0x7f21641430c0 2024-03-25T19:44:58.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:58.997+0000 7f2169f65640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:58.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:58.997+0000 7f2169f65640 10 monclient(hunting): authenticate will time out at 5963.168039s 2024-03-25T19:44:59.007 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.007+0000 7f21637fe640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f21640a98f0 0x7f21640a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:43942/0 (socket says 172.21.15.44:43942) 2024-03-25T19:44:59.007 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.007+0000 7f21637fe640 1 -- 172.21.15.44:0/3013084335 learned_addr learned my addr 172.21.15.44:0/3013084335 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:44:59.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.014+0000 7f21627fc640 1 -- 172.21.15.44:0/3013084335 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110452419 0 0) 0x7f21640b28a0 con 0x7f21640a98f0 2024-03-25T19:44:59.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.014+0000 7f21627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:59.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.014+0000 7f21627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:59.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.014+0000 7f21627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:59.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.014+0000 7f21627fc640 10 monclient(hunting): my global_id is 4651 2024-03-25T19:44:59.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.014+0000 7f21627fc640 1 -- 172.21.15.44:0/3013084335 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f214c003520 con 0x7f21640a98f0 2024-03-25T19:44:59.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.026+0000 7f21627fc640 1 -- 172.21.15.44:0/3013084335 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2922317330 0 0) 0x7f2164142520 con 0x7f216413f710 2024-03-25T19:44:59.027 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.026+0000 7f21627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:59.027 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.026+0000 7f21627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:59.027 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.026+0000 7f21627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:59.027 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.026+0000 7f21627fc640 10 monclient(hunting): my global_id is 4665 2024-03-25T19:44:59.027 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.026+0000 7f21627fc640 1 -- 172.21.15.44:0/3013084335 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21640b28a0 con 0x7f216413f710 2024-03-25T19:44:59.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.033+0000 7f21627fc640 1 -- 172.21.15.44:0/3013084335 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213578073 0 0) 0x7f21640b2a30 con 0x7f21641430c0 2024-03-25T19:44:59.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.033+0000 7f21627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:59.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.033+0000 7f21627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:59.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.033+0000 7f21627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:59.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.033+0000 7f21627fc640 10 monclient(hunting): my global_id is 4643 2024-03-25T19:44:59.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.033+0000 7f21627fc640 1 -- 172.21.15.44:0/3013084335 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2164142520 con 0x7f21641430c0 2024-03-25T19:44:59.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.039+0000 7f21627fc640 1 -- 172.21.15.44:0/3013084335 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1211199863 0 0) 0x7f21640b28a0 con 0x7f216413f710 2024-03-25T19:44:59.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.039+0000 7f21627fc640 1 -- 172.21.15.44:0/3013084335 >> v1:172.21.15.44:6790/0 conn(0x7f21641430c0 legacy=0x7f216414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:59.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.039+0000 7f21627fc640 1 -- 172.21.15.44:0/3013084335 >> v1:172.21.15.111:6789/0 conn(0x7f21640a98f0 legacy=0x7f21640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:59.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.039+0000 7f21627fc640 10 monclient: _finish_hunting 0 2024-03-25T19:44:59.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.039+0000 7f21627fc640 1 monclient: found mon.noname-a 2024-03-25T19:44:59.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.039+0000 7f21627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:59.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.039+0000 7f21627fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:44:59.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.039+0000 7f21627fc640 1 -- 172.21.15.44:0/3013084335 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21640b44c0 con 0x7f216413f710 2024-03-25T19:44:59.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.039+0000 7f21627fc640 10 monclient: _finish_auth 0 2024-03-25T19:44:59.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.039+0000 7f21627fc640 10 monclient: _check_auth_tickets 2024-03-25T19:44:59.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.039+0000 7f21627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:59.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.039+0000 7f21627fc640 1 -- 172.21.15.44:0/3013084335 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f2158001670 con 0x7f216413f710 2024-03-25T19:44:59.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.039+0000 7f2169f65640 5 monclient: authenticate success, global_id 4665 2024-03-25T19:44:59.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.039+0000 7f2169f65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:59.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.039+0000 7f21627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:59.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.039+0000 7f21627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:59.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.039+0000 7f21627fc640 10 monclient: dump: 2024-03-25T19:44:59.041 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:59.041 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:59.041 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:59.041 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:59.041 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:59.041 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:59.041 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:59.041 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:59.041 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:59.041 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:59.041 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.039+0000 7f2169f65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:59.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.062+0000 7f21627fc640 1 -- 172.21.15.44:0/3013084335 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21580017d0 con 0x7f216413f710 2024-03-25T19:44:59.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.062+0000 7f21627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:59.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.062+0000 7f21627fc640 1 -- 172.21.15.44:0/3013084335 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f2158001be0 con 0x7f216413f710 2024-03-25T19:44:59.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.062+0000 7f2169f65640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:59.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.062+0000 7f2169f65640 4 set_mon_vals no callback set 2024-03-25T19:44:59.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.062+0000 7f21627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:59.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.062+0000 7f21627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:59.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.062+0000 7f21627fc640 10 monclient: dump: 2024-03-25T19:44:59.063 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:59.063 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:59.063 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:59.063 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:59.063 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:59.063 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:59.063 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:59.063 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:59.063 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:59.064 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:59.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.063+0000 7f2169f65640 10 monclient: shutdown 2024-03-25T19:44:59.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.063+0000 7f2169f65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:59.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.063+0000 7f2169f65640 1 -- 172.21.15.44:0/3013084335 >> v1:172.21.15.44:6789/0 conn(0x7f216413f710 legacy=0x7f21641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:59.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.063+0000 7f2169f65640 1 -- 172.21.15.44:0/3013084335 shutdown_connections 2024-03-25T19:44:59.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.063+0000 7f2169f65640 1 -- 172.21.15.44:0/3013084335 >> 172.21.15.44:0/3013084335 conn(0x7f21640b7ea0 msgr2=0x7f21640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:59.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.063+0000 7f2169f65640 1 -- 172.21.15.44:0/3013084335 shutdown_connections 2024-03-25T19:44:59.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.063+0000 7f2169f65640 1 -- 172.21.15.44:0/3013084335 wait complete. 2024-03-25T19:44:59.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.064+0000 7f2169f65640 10 monclient: build_initial_monmap 2024-03-25T19:44:59.065 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.064+0000 7f2169f65640 10 monclient: monmap: 2024-03-25T19:44:59.066 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:44:59.066 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:59.066 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:44:59.066 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:44:59.066 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:59.066 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:44:59.066 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:44:59.066 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:44:59.066 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:44:59.066 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:59.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.064+0000 7f2169f65640 1 librados: starting msgr at 2024-03-25T19:44:59.066 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.064+0000 7f2169f65640 1 librados: starting objecter 2024-03-25T19:44:59.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.064+0000 7f2169f65640 1 Processor -- start 2024-03-25T19:44:59.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.065+0000 7f2169f65640 1 -- start start 2024-03-25T19:44:59.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.065+0000 7f2169f65640 1 librados: setting wanted keys 2024-03-25T19:44:59.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.065+0000 7f2169f65640 1 librados: calling monclient init 2024-03-25T19:44:59.067 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.065+0000 7f2169f65640 10 monclient: init 2024-03-25T19:44:59.068 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.065+0000 7f2169f65640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:59.068 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.065+0000 7f2169f65640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:44:59.068 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.065+0000 7f2169f65640 10 monclient(hunting): picked mon.noname-a con 0x7f21640a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:44:59.069 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.065+0000 7f2169f65640 10 monclient(hunting): picked mon.noname-b con 0x7f216413f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:44:59.069 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.065+0000 7f2169f65640 10 monclient(hunting): picked mon.noname-c con 0x7f21641430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:44:59.069 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.065+0000 7f2169f65640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2164136750 con 0x7f21640a98f0 2024-03-25T19:44:59.069 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.065+0000 7f2169f65640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21641dac30 con 0x7f216413f710 2024-03-25T19:44:59.069 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.065+0000 7f2169f65640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21641399c0 con 0x7f21641430c0 2024-03-25T19:44:59.069 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.065+0000 7f2169f65640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:59.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.065+0000 7f2169f65640 10 monclient(hunting): authenticate will time out at 5963.236628s 2024-03-25T19:44:59.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.068+0000 7f2163fff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f21641430c0 0x7f21641d4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:43528/0 (socket says 172.21.15.44:43528) 2024-03-25T19:44:59.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.068+0000 7f2163fff640 1 -- 172.21.15.44:0/3935570517 learned_addr learned my addr 172.21.15.44:0/3935570517 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:44:59.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.071+0000 7f214bfff640 10 client.?.objecter ms_handle_connect 0x7f21641430c0 2024-03-25T19:44:59.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.071+0000 7f214bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:59.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.071+0000 7f214bfff640 10 client.?.objecter ms_handle_connect 0x7f216413f710 2024-03-25T19:44:59.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.071+0000 7f214bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:59.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.075+0000 7f214bfff640 1 -- 172.21.15.44:0/3935570517 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644717170 0 0) 0x7f21641399c0 con 0x7f21641430c0 2024-03-25T19:44:59.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.075+0000 7f214bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:59.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.075+0000 7f214bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:59.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.075+0000 7f214bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:59.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.075+0000 7f214bfff640 10 monclient(hunting): my global_id is 4646 2024-03-25T19:44:59.075 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.075+0000 7f214bfff640 1 -- 172.21.15.44:0/3935570517 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2150004eb0 con 0x7f21641430c0 2024-03-25T19:44:59.079 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.079+0000 7f214bfff640 1 -- 172.21.15.44:0/3935570517 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081486854 0 0) 0x7f21641dac30 con 0x7f216413f710 2024-03-25T19:44:59.079 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.079+0000 7f214bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:59.079 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.079+0000 7f214bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:59.079 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.079+0000 7f214bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:59.079 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.079+0000 7f214bfff640 10 monclient(hunting): my global_id is 4654 2024-03-25T19:44:59.079 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.079+0000 7f214bfff640 1 -- 172.21.15.44:0/3935570517 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21641399c0 con 0x7f216413f710 2024-03-25T19:44:59.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.084+0000 7f214bfff640 10 client.?.objecter ms_handle_connect 0x7f21640a98f0 2024-03-25T19:44:59.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.084+0000 7f214bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:59.089 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.088+0000 7f214bfff640 1 -- 172.21.15.44:0/3935570517 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1086918904 0 0) 0x7f2150004eb0 con 0x7f21641430c0 2024-03-25T19:44:59.089 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.088+0000 7f214bfff640 1 -- 172.21.15.44:0/3935570517 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21641dac30 con 0x7f21641430c0 2024-03-25T19:44:59.089 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.088+0000 7f214bfff640 1 -- 172.21.15.44:0/3935570517 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f215c001c70 con 0x7f21641430c0 2024-03-25T19:44:59.089 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.088+0000 7f214bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:59.089 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.088+0000 7f214bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:59.089 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.088+0000 7f214bfff640 10 monclient(hunting): dump: 2024-03-25T19:44:59.089 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:59.089 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:59.089 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:59.089 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:59.089 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:59.089 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:59.090 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:59.090 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:59.090 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:59.090 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:59.090 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.088+0000 7f214bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:59.090 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.089+0000 7f214bfff640 1 -- 172.21.15.44:0/3935570517 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1854997749 0 0) 0x7f21641399c0 con 0x7f216413f710 2024-03-25T19:44:59.090 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.089+0000 7f214bfff640 1 -- 172.21.15.44:0/3935570517 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2150004eb0 con 0x7f216413f710 2024-03-25T19:44:59.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.089+0000 7f214bfff640 1 -- 172.21.15.44:0/3935570517 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f2158001b20 con 0x7f216413f710 2024-03-25T19:44:59.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.089+0000 7f214bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:59.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.089+0000 7f214bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:59.091 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.089+0000 7f214bfff640 10 monclient(hunting): dump: 2024-03-25T19:44:59.091 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:59.091 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:59.091 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:59.091 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:59.091 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:59.091 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:59.091 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:59.091 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:59.091 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:59.091 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:59.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.096+0000 7f214bfff640 1 -- 172.21.15.44:0/3935570517 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1319847240 0 0) 0x7f2150004eb0 con 0x7f216413f710 2024-03-25T19:44:59.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.096+0000 7f214bfff640 1 -- 172.21.15.44:0/3935570517 >> v1:172.21.15.44:6790/0 conn(0x7f21641430c0 legacy=0x7f21641d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:59.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.096+0000 7f214bfff640 1 -- 172.21.15.44:0/3935570517 >> v1:172.21.15.44:6789/0 conn(0x7f21640a98f0 legacy=0x7f21641386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:59.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.096+0000 7f214bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:44:59.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.096+0000 7f214bfff640 1 monclient: found mon.b 2024-03-25T19:44:59.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.096+0000 7f214bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:59.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.096+0000 7f214bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:44:59.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.096+0000 7f214bfff640 1 -- 172.21.15.44:0/3935570517 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21641db860 con 0x7f216413f710 2024-03-25T19:44:59.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.096+0000 7f214bfff640 10 monclient: _finish_auth 0 2024-03-25T19:44:59.097 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.096+0000 7f214bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:44:59.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.096+0000 7f214bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:59.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.096+0000 7f2169f65640 5 monclient: authenticate success, global_id 4654 2024-03-25T19:44:59.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.096+0000 7f2169f65640 10 monclient: _renew_subs 2024-03-25T19:44:59.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.096+0000 7f2169f65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:44:59.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.096+0000 7f2169f65640 1 -- 172.21.15.44:0/3935570517 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21641d41e0 con 0x7f216413f710 2024-03-25T19:44:59.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.096+0000 7f2169f65640 10 client.4654.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:59.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.096+0000 7f2169f65640 10 monclient: _renew_subs 2024-03-25T19:44:59.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.096+0000 7f2169f65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:44:59.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.096+0000 7f2169f65640 1 -- 172.21.15.44:0/3935570517 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21641d5150 con 0x7f216413f710 2024-03-25T19:44:59.098 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.096+0000 7f2169f65640 1 librados: init done 2024-03-25T19:44:59.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.097+0000 7f2169f65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:59.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.097+0000 7f2169f65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:59.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.097+0000 7f2169f65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:44:59.099 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.097+0000 7f2169f65640 1 -- 172.21.15.44:0/3935570517 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21641daa90 con 0x7f216413f710 2024-03-25T19:44:59.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.364+0000 7f214bfff640 1 -- 172.21.15.44:0/3935570517 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21580038a0 con 0x7f216413f710 2024-03-25T19:44:59.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.364+0000 7f214bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:59.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.364+0000 7f214bfff640 1 -- 172.21.15.44:0/3935570517 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f2158003a10 con 0x7f216413f710 2024-03-25T19:44:59.366 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.364+0000 7f214bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:59.366 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.364+0000 7f214bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:59.366 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.364+0000 7f214bfff640 10 monclient: dump: 2024-03-25T19:44:59.366 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:44:59.366 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:44:59.367 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:59.367 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:44:59.367 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:59.367 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:44:59.367 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:44:59.367 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:44:59.367 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:44:59.368 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:44:59.368 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.364+0000 7f2161ffb640 4 set_mon_vals no callback set 2024-03-25T19:44:59.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.365+0000 7f214bfff640 1 -- 172.21.15.44:0/3935570517 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f21580199e0 con 0x7f216413f710 2024-03-25T19:44:59.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.366+0000 7f21637fe640 10 monclient: get_auth_request con 0x7f215006de80 auth_method 0 2024-03-25T19:44:59.369 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.366+0000 7f214bfff640 10 client.4654.objecter ms_handle_connect 0x7f215006de80 2024-03-25T19:44:59.373 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.373+0000 7f214bfff640 1 -- 172.21.15.44:0/3935570517 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f2158053520 con 0x7f216413f710 2024-03-25T19:44:59.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.373+0000 7f214bfff640 10 client.4654.objecter ms_dispatch 0x7f2164000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:59.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.373+0000 7f214bfff640 3 client.4654.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:59.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.373+0000 7f214bfff640 3 client.4654.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:59.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.373+0000 7f214bfff640 20 client.4654.objecter dump_active .. 0 homeless 2024-03-25T19:44:59.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.373+0000 7f214bfff640 1 -- 172.21.15.44:0/3935570517 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f215804f030 con 0x7f216413f710 2024-03-25T19:44:59.374 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.373+0000 7f214bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:59.375 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.373+0000 7f214bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:59.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.631+0000 7f2169f65640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:59.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.631+0000 7f2169f65640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:59.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.631+0000 7f2169f65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:44:59.632 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.631+0000 7f2169f65640 1 -- 172.21.15.44:0/3935570517 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21641db9f0 con 0x7f216413f710 2024-03-25T19:44:59.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.651+0000 7f214bfff640 1 -- 172.21.15.44:0/3935570517 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f21641db9f0 con 0x7f216413f710 2024-03-25T19:44:59.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.651+0000 7f214bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:59.652 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.651+0000 7f214bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:59.652 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:44:59.652 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:59.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.654+0000 7f2169f65640 10 librados: watch_flush enter 2024-03-25T19:44:59.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.654+0000 7f2169f65640 10 librados: watch_flush exit 2024-03-25T19:44:59.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.654+0000 7f2169f65640 20 client.4654.objecter shutdown clearing up homeless session... 2024-03-25T19:44:59.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.654+0000 7f2169f65640 10 client.4654.objecter successfully canceled tick 2024-03-25T19:44:59.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.654+0000 7f2169f65640 1 -- 172.21.15.44:0/3935570517 >> v1:172.21.15.111:6800/3883215793 conn(0x7f215006de80 legacy=0x7f2150070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:59.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.654+0000 7f2169f65640 10 monclient: shutdown 2024-03-25T19:44:59.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.654+0000 7f2169f65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:59.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.654+0000 7f2169f65640 1 -- 172.21.15.44:0/3935570517 >> v1:172.21.15.111:6789/0 conn(0x7f216413f710 legacy=0x7f21641db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:59.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.654+0000 7f2169f65640 1 -- 172.21.15.44:0/3935570517 shutdown_connections 2024-03-25T19:44:59.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.654+0000 7f2169f65640 1 -- 172.21.15.44:0/3935570517 >> 172.21.15.44:0/3935570517 conn(0x7f21640b7ea0 msgr2=0x7f2164074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:59.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.654+0000 7f2169f65640 1 -- 172.21.15.44:0/3935570517 shutdown_connections 2024-03-25T19:44:59.655 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.654+0000 7f2169f65640 1 -- 172.21.15.44:0/3935570517 wait complete. 2024-03-25T19:44:59.656 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:44:59.655+0000 7f2169f65640 1 librados: shutdown 2024-03-25T19:44:59.666 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:05.556 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:09:56.457 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:05.565 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:09:56.467 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:05.584 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:09:56.486 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:05.613 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:09:56.492 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:05.626 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:09:56.527 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:05.627 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:09:56.507 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:05.653 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:09:56.532 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:05.655 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:09:56.557 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:05.686 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:09:56.588 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:06.473 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:06.589 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.589+0000 7fdc7e8f6640 10 monclient: get_monmap_and_config 2024-03-25T19:45:06.589 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.589+0000 7fdc7e8f6640 10 monclient: build_initial_monmap 2024-03-25T19:45:06.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.589+0000 7fdc7e8f6640 10 monclient: monmap: 2024-03-25T19:45:06.590 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:45:06.590 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:06.590 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:45:06.590526+0000 2024-03-25T19:45:06.590 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:45:06.590526+0000 2024-03-25T19:45:06.590 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:06.590 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:06.590 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:45:06.590 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:45:06.590 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:45:06.590 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:06.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.590+0000 7fdc7e8f6640 1 Processor -- start 2024-03-25T19:45:06.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.590+0000 7fdc7e8f6640 1 -- start start 2024-03-25T19:45:06.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.590+0000 7fdc7e8f6640 10 monclient: init 2024-03-25T19:45:06.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.590+0000 7fdc7e8f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:06.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.590+0000 7fdc7e8f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:45:06.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.590+0000 7fdc7e8f6640 10 monclient(hunting): picked mon.noname-b con 0x7fdc780b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T19:45:06.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.590+0000 7fdc7e8f6640 10 monclient(hunting): picked mon.noname-a con 0x7fdc780b1730 addr v1:172.21.15.44:6789/0 2024-03-25T19:45:06.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.590+0000 7fdc7e8f6640 10 monclient(hunting): picked mon.noname-c con 0x7fdc7814be90 addr v1:172.21.15.111:6789/0 2024-03-25T19:45:06.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.591+0000 7fdc7e8f6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc780af940 con 0x7fdc780b1730 2024-03-25T19:45:06.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.591+0000 7fdc7e8f6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc780b0a70 con 0x7fdc7814be90 2024-03-25T19:45:06.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.591+0000 7fdc7e8f6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc780b0d50 con 0x7fdc780b4d60 2024-03-25T19:45:06.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.591+0000 7fdc7e8f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:06.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.591+0000 7fdc7e8f6640 10 monclient(hunting): authenticate will time out at 5970.761881s 2024-03-25T19:45:06.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.592+0000 7fdc77fff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fdc780b4d60 0x7fdc7813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:48116/0 (socket says 172.21.15.44:48116) 2024-03-25T19:45:06.593 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.592+0000 7fdc77fff640 1 -- 172.21.15.44:0/1137383053 learned_addr learned my addr 172.21.15.44:0/1137383053 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:45:06.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.663+0000 7fdc76ffd640 1 -- 172.21.15.44:0/1137383053 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852735669 0 0) 0x7fdc780b0d50 con 0x7fdc780b4d60 2024-03-25T19:45:06.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.663+0000 7fdc76ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:06.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.663+0000 7fdc76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:06.663 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.663+0000 7fdc76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:06.664 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.663+0000 7fdc76ffd640 10 monclient(hunting): my global_id is 4649 2024-03-25T19:45:06.664 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.663+0000 7fdc76ffd640 1 -- 172.21.15.44:0/1137383053 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc60003520 con 0x7fdc780b4d60 2024-03-25T19:45:06.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.676+0000 7fdc76ffd640 1 -- 172.21.15.44:0/1137383053 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358235195 0 0) 0x7fdc780af940 con 0x7fdc780b1730 2024-03-25T19:45:06.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.676+0000 7fdc76ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:06.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.676+0000 7fdc76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:06.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.676+0000 7fdc76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:06.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.676+0000 7fdc76ffd640 10 monclient(hunting): my global_id is 4671 2024-03-25T19:45:06.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.676+0000 7fdc76ffd640 1 -- 172.21.15.44:0/1137383053 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc780b0d50 con 0x7fdc780b1730 2024-03-25T19:45:06.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.678+0000 7fdc76ffd640 1 -- 172.21.15.44:0/1137383053 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893375794 0 0) 0x7fdc780b0a70 con 0x7fdc7814be90 2024-03-25T19:45:06.678 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.678+0000 7fdc76ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:06.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.678+0000 7fdc76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:06.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.678+0000 7fdc76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:06.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.678+0000 7fdc76ffd640 10 monclient(hunting): my global_id is 4657 2024-03-25T19:45:06.679 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.678+0000 7fdc76ffd640 1 -- 172.21.15.44:0/1137383053 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc780af940 con 0x7fdc7814be90 2024-03-25T19:45:06.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.686+0000 7fdc76ffd640 1 -- 172.21.15.44:0/1137383053 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 300321685 0 0) 0x7fdc60003520 con 0x7fdc780b4d60 2024-03-25T19:45:06.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.686+0000 7fdc76ffd640 1 -- 172.21.15.44:0/1137383053 >> v1:172.21.15.111:6789/0 conn(0x7fdc7814be90 legacy=0x7fdc7814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:06.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.686+0000 7fdc76ffd640 1 -- 172.21.15.44:0/1137383053 >> v1:172.21.15.44:6789/0 conn(0x7fdc780b1730 legacy=0x7fdc780b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:06.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.686+0000 7fdc76ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:45:06.687 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.686+0000 7fdc76ffd640 1 monclient: found mon.noname-b 2024-03-25T19:45:06.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.686+0000 7fdc76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:06.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.686+0000 7fdc76ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:45:06.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.686+0000 7fdc76ffd640 1 -- 172.21.15.44:0/1137383053 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc780b1030 con 0x7fdc780b4d60 2024-03-25T19:45:06.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.686+0000 7fdc76ffd640 10 monclient: _finish_auth 0 2024-03-25T19:45:06.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.686+0000 7fdc76ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:45:06.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.686+0000 7fdc76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:06.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.686+0000 7fdc76ffd640 1 -- 172.21.15.44:0/1137383053 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fdc68001690 con 0x7fdc780b4d60 2024-03-25T19:45:06.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.686+0000 7fdc76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:06.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.686+0000 7fdc76ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:06.688 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.686+0000 7fdc76ffd640 10 monclient: dump: 2024-03-25T19:45:06.688 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:06.688 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:06.688 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:06.688 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:06.688 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:06.689 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:06.689 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:06.689 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:06.689 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:06.689 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:06.689 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.686+0000 7fdc7e8f6640 5 monclient: authenticate success, global_id 4649 2024-03-25T19:45:06.689 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.686+0000 7fdc7e8f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:06.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.698+0000 7fdc76ffd640 1 -- 172.21.15.44:0/1137383053 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc680017f0 con 0x7fdc780b4d60 2024-03-25T19:45:06.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.698+0000 7fdc76ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:06.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.698+0000 7fdc76ffd640 1 -- 172.21.15.44:0/1137383053 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fdc68001c00 con 0x7fdc780b4d60 2024-03-25T19:45:06.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.698+0000 7fdc7e8f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:06.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.698+0000 7fdc7e8f6640 4 set_mon_vals no callback set 2024-03-25T19:45:06.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.698+0000 7fdc76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:06.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.698+0000 7fdc76ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:06.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.698+0000 7fdc76ffd640 10 monclient: dump: 2024-03-25T19:45:06.698 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:06.699 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:06.699 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:06.699 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:06.699 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:06.699 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:06.699 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:06.699 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:06.699 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:06.699 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:06.701 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.698+0000 7fdc7e8f6640 10 monclient: shutdown 2024-03-25T19:45:06.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.698+0000 7fdc7e8f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:06.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.698+0000 7fdc7e8f6640 1 -- 172.21.15.44:0/1137383053 >> v1:172.21.15.44:6790/0 conn(0x7fdc780b4d60 legacy=0x7fdc7813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:06.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.698+0000 7fdc7e8f6640 1 -- 172.21.15.44:0/1137383053 shutdown_connections 2024-03-25T19:45:06.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.698+0000 7fdc7e8f6640 1 -- 172.21.15.44:0/1137383053 >> 172.21.15.44:0/1137383053 conn(0x7fdc780617a0 msgr2=0x7fdc78061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:06.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.698+0000 7fdc7e8f6640 1 -- 172.21.15.44:0/1137383053 shutdown_connections 2024-03-25T19:45:06.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.698+0000 7fdc7e8f6640 1 -- 172.21.15.44:0/1137383053 wait complete. 2024-03-25T19:45:06.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.699+0000 7fdc7e8f6640 10 monclient: build_initial_monmap 2024-03-25T19:45:06.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.699+0000 7fdc7e8f6640 10 monclient: monmap: 2024-03-25T19:45:06.702 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:45:06.702 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:06.702 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:45:06.702 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:45:06.702 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:06.702 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:45:06.703 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:45:06.703 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:45:06.703 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:45:06.703 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:06.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.699+0000 7fdc7e8f6640 1 librados: starting msgr at 2024-03-25T19:45:06.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.699+0000 7fdc7e8f6640 1 librados: starting objecter 2024-03-25T19:45:06.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.700+0000 7fdc7e8f6640 1 Processor -- start 2024-03-25T19:45:06.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.700+0000 7fdc7e8f6640 1 -- start start 2024-03-25T19:45:06.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.700+0000 7fdc7e8f6640 1 librados: setting wanted keys 2024-03-25T19:45:06.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.700+0000 7fdc7e8f6640 1 librados: calling monclient init 2024-03-25T19:45:06.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.700+0000 7fdc7e8f6640 10 monclient: init 2024-03-25T19:45:06.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.700+0000 7fdc7e8f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:06.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.700+0000 7fdc7e8f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:45:06.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.700+0000 7fdc7e8f6640 10 monclient(hunting): picked mon.noname-b con 0x7fdc780b1730 addr v1:172.21.15.111:6789/0 2024-03-25T19:45:06.704 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.700+0000 7fdc7e8f6640 10 monclient(hunting): picked mon.noname-c con 0x7fdc780b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T19:45:06.704 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.700+0000 7fdc7e8f6640 10 monclient(hunting): picked mon.noname-a con 0x7fdc7814be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:45:06.704 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.700+0000 7fdc7e8f6640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc781361d0 con 0x7fdc7814be90 2024-03-25T19:45:06.704 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.700+0000 7fdc7e8f6640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc78136c40 con 0x7fdc780b1730 2024-03-25T19:45:06.704 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.700+0000 7fdc7e8f6640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc78136f20 con 0x7fdc780b4d60 2024-03-25T19:45:06.704 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.700+0000 7fdc7e8f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:06.704 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.700+0000 7fdc7e8f6640 10 monclient(hunting): authenticate will time out at 5970.871788s 2024-03-25T19:45:06.704 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.702+0000 7fdc77fff640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fdc780b1730 0x7fdc7812e700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:48794/0 (socket says 172.21.15.44:48794) 2024-03-25T19:45:06.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.702+0000 7fdc77fff640 1 -- 172.21.15.44:0/999933321 learned_addr learned my addr 172.21.15.44:0/999933321 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:45:06.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.702+0000 7fdc74ff9640 10 client.?.objecter ms_handle_connect 0x7fdc780b1730 2024-03-25T19:45:06.705 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.702+0000 7fdc74ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:06.711 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.711+0000 7fdc74ff9640 10 client.?.objecter ms_handle_connect 0x7fdc780b4d60 2024-03-25T19:45:06.711 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.711+0000 7fdc74ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:06.716 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.716+0000 7fdc74ff9640 10 client.?.objecter ms_handle_connect 0x7fdc7814be90 2024-03-25T19:45:06.716 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.716+0000 7fdc74ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:06.739 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.739+0000 7fdc74ff9640 1 -- 172.21.15.44:0/999933321 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898837489 0 0) 0x7fdc781361d0 con 0x7fdc7814be90 2024-03-25T19:45:06.739 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.739+0000 7fdc74ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:06.739 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.739+0000 7fdc74ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:06.739 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.739+0000 7fdc74ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:06.739 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.739+0000 7fdc74ff9640 10 monclient(hunting): my global_id is 4674 2024-03-25T19:45:06.740 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.739+0000 7fdc74ff9640 1 -- 172.21.15.44:0/999933321 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc64004eb0 con 0x7fdc7814be90 2024-03-25T19:45:06.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.809+0000 7fdc74ff9640 1 -- 172.21.15.44:0/999933321 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946656987 0 0) 0x7fdc78136f20 con 0x7fdc780b4d60 2024-03-25T19:45:06.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.810+0000 7fdc74ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:06.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.810+0000 7fdc74ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:06.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.810+0000 7fdc74ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:06.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.810+0000 7fdc74ff9640 10 monclient(hunting): my global_id is 4652 2024-03-25T19:45:06.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.810+0000 7fdc74ff9640 1 -- 172.21.15.44:0/999933321 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc781361d0 con 0x7fdc780b4d60 2024-03-25T19:45:06.814 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.814+0000 7fdc74ff9640 1 -- 172.21.15.44:0/999933321 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1635511895 0 0) 0x7fdc64004eb0 con 0x7fdc7814be90 2024-03-25T19:45:06.815 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.814+0000 7fdc74ff9640 1 -- 172.21.15.44:0/999933321 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc78136f20 con 0x7fdc7814be90 2024-03-25T19:45:06.815 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.814+0000 7fdc74ff9640 1 -- 172.21.15.44:0/999933321 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fdc70001bd0 con 0x7fdc7814be90 2024-03-25T19:45:06.815 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.814+0000 7fdc74ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:06.815 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.814+0000 7fdc74ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:06.815 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.814+0000 7fdc74ff9640 10 monclient(hunting): dump: 2024-03-25T19:45:06.815 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:06.815 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:06.815 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:06.815 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:06.815 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:06.815 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:06.815 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:06.815 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:06.815 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:06.815 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:06.816 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.814+0000 7fdc74ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:06.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.832+0000 7fdc74ff9640 1 -- 172.21.15.44:0/999933321 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 653202623 0 0) 0x7fdc781361d0 con 0x7fdc780b4d60 2024-03-25T19:45:06.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.832+0000 7fdc74ff9640 1 -- 172.21.15.44:0/999933321 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc64004eb0 con 0x7fdc780b4d60 2024-03-25T19:45:06.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.832+0000 7fdc74ff9640 1 -- 172.21.15.44:0/999933321 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fdc6c001970 con 0x7fdc780b4d60 2024-03-25T19:45:06.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.832+0000 7fdc74ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:06.832 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.832+0000 7fdc74ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:06.833 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.832+0000 7fdc74ff9640 10 monclient(hunting): dump: 2024-03-25T19:45:06.833 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:06.833 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:06.833 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:06.833 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:06.833 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:06.833 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:06.833 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:06.833 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:06.833 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:06.833 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:06.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.838+0000 7fdc74ff9640 1 -- 172.21.15.44:0/999933321 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 767758006 0 0) 0x7fdc78136f20 con 0x7fdc7814be90 2024-03-25T19:45:06.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.838+0000 7fdc74ff9640 1 -- 172.21.15.44:0/999933321 >> v1:172.21.15.44:6790/0 conn(0x7fdc780b4d60 legacy=0x7fdc7812cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:06.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.838+0000 7fdc74ff9640 1 -- 172.21.15.44:0/999933321 >> v1:172.21.15.111:6789/0 conn(0x7fdc780b1730 legacy=0x7fdc7812e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:06.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.838+0000 7fdc74ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:45:06.839 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.838+0000 7fdc74ff9640 1 monclient: found mon.a 2024-03-25T19:45:06.840 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.838+0000 7fdc74ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:06.840 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.838+0000 7fdc74ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:45:06.840 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.838+0000 7fdc74ff9640 1 -- 172.21.15.44:0/999933321 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc78137200 con 0x7fdc7814be90 2024-03-25T19:45:06.840 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.838+0000 7fdc74ff9640 10 monclient: _finish_auth 0 2024-03-25T19:45:06.840 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.838+0000 7fdc74ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:45:06.840 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.838+0000 7fdc74ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:06.840 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.838+0000 7fdc7e8f6640 5 monclient: authenticate success, global_id 4674 2024-03-25T19:45:06.840 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.838+0000 7fdc7e8f6640 10 monclient: _renew_subs 2024-03-25T19:45:06.840 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.838+0000 7fdc7e8f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:45:06.840 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.838+0000 7fdc7e8f6640 1 -- 172.21.15.44:0/999933321 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc78135cb0 con 0x7fdc7814be90 2024-03-25T19:45:06.840 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.838+0000 7fdc7e8f6640 10 client.4674.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:06.840 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.838+0000 7fdc7e8f6640 10 monclient: _renew_subs 2024-03-25T19:45:06.840 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.838+0000 7fdc7e8f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:45:06.840 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.838+0000 7fdc7e8f6640 1 -- 172.21.15.44:0/999933321 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc7813e930 con 0x7fdc7814be90 2024-03-25T19:45:06.841 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.839+0000 7fdc7e8f6640 1 librados: init done 2024-03-25T19:45:06.841 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.839+0000 7fdc7e8f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:06.841 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.840+0000 7fdc7e8f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:06.841 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.840+0000 7fdc7e8f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:45:06.841 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:06.840+0000 7fdc7e8f6640 1 -- 172.21.15.44:0/999933321 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc781401f0 con 0x7fdc7814be90 2024-03-25T19:45:07.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.175+0000 7fdc74ff9640 1 -- 172.21.15.44:0/999933321 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc70002060 con 0x7fdc7814be90 2024-03-25T19:45:07.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.175+0000 7fdc74ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:07.175 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.175+0000 7fdc74ff9640 1 -- 172.21.15.44:0/999933321 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fdc70003b30 con 0x7fdc7814be90 2024-03-25T19:45:07.176 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.175+0000 7fdc74ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:07.176 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.175+0000 7fdc74ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:07.176 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.175+0000 7fdc74ff9640 10 monclient: dump: 2024-03-25T19:45:07.176 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:07.176 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:07.176 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:07.176 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:07.176 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:07.176 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:07.176 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:07.176 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:07.176 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:07.176 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:07.176 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.175+0000 7fdc767fc640 4 set_mon_vals no callback set 2024-03-25T19:45:07.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.176+0000 7fdc74ff9640 1 -- 172.21.15.44:0/999933321 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7fdc70019b00 con 0x7fdc7814be90 2024-03-25T19:45:07.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.176+0000 7fdc77fff640 10 monclient: get_auth_request con 0x7fdc6406ded0 auth_method 0 2024-03-25T19:45:07.177 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.176+0000 7fdc74ff9640 10 client.4674.objecter ms_handle_connect 0x7fdc6406ded0 2024-03-25T19:45:07.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.181+0000 7fdc74ff9640 1 -- 172.21.15.44:0/999933321 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7fdc700537b0 con 0x7fdc7814be90 2024-03-25T19:45:07.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.181+0000 7fdc74ff9640 10 client.4674.objecter ms_dispatch 0x7fdc78000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:07.181 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.181+0000 7fdc74ff9640 3 client.4674.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:07.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.181+0000 7fdc74ff9640 3 client.4674.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:07.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.181+0000 7fdc74ff9640 20 client.4674.objecter dump_active .. 0 homeless 2024-03-25T19:45:07.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.181+0000 7fdc74ff9640 1 -- 172.21.15.44:0/999933321 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdc7004f0f0 con 0x7fdc7814be90 2024-03-25T19:45:07.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.181+0000 7fdc74ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:07.182 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.181+0000 7fdc74ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:07.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.436+0000 7fdc7e8f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:07.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.436+0000 7fdc7e8f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:07.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.436+0000 7fdc7e8f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:45:07.436 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.436+0000 7fdc7e8f6640 1 -- 172.21.15.44:0/999933321 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc78135ef0 con 0x7fdc7814be90 2024-03-25T19:45:07.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.491+0000 7fdc74ff9640 1 -- 172.21.15.44:0/999933321 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7fdc78135ef0 con 0x7fdc7814be90 2024-03-25T19:45:07.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.491+0000 7fdc74ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:07.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.491+0000 7fdc74ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:07.492 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:45:07.492 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:07.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.493+0000 7fdc7e8f6640 10 librados: watch_flush enter 2024-03-25T19:45:07.494 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.493+0000 7fdc7e8f6640 10 librados: watch_flush exit 2024-03-25T19:45:07.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.493+0000 7fdc7e8f6640 20 client.4674.objecter shutdown clearing up homeless session... 2024-03-25T19:45:07.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.493+0000 7fdc7e8f6640 10 client.4674.objecter successfully canceled tick 2024-03-25T19:45:07.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.493+0000 7fdc7e8f6640 1 -- 172.21.15.44:0/999933321 >> v1:172.21.15.111:6800/3883215793 conn(0x7fdc6406ded0 legacy=0x7fdc64070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:07.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.493+0000 7fdc7e8f6640 10 monclient: shutdown 2024-03-25T19:45:07.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.493+0000 7fdc7e8f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:07.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.493+0000 7fdc7e8f6640 1 -- 172.21.15.44:0/999933321 >> v1:172.21.15.44:6789/0 conn(0x7fdc7814be90 legacy=0x7fdc781e33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:07.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.494+0000 7fdc7e8f6640 1 -- 172.21.15.44:0/999933321 shutdown_connections 2024-03-25T19:45:07.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.494+0000 7fdc7e8f6640 1 -- 172.21.15.44:0/999933321 >> 172.21.15.44:0/999933321 conn(0x7fdc780617a0 msgr2=0x7fdc780b6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:07.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.494+0000 7fdc7e8f6640 1 -- 172.21.15.44:0/999933321 shutdown_connections 2024-03-25T19:45:07.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.494+0000 7fdc7e8f6640 1 -- 172.21.15.44:0/999933321 wait complete. 2024-03-25T19:45:07.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:07.494+0000 7fdc7e8f6640 1 librados: shutdown 2024-03-25T19:45:07.505 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:14.311 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:14.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.429+0000 7f321c148640 10 monclient: get_monmap_and_config 2024-03-25T19:45:14.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.429+0000 7f321c148640 10 monclient: build_initial_monmap 2024-03-25T19:45:14.429 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.429+0000 7f321c148640 10 monclient: monmap: 2024-03-25T19:45:14.429 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:45:14.429 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:14.429 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:45:14.430179+0000 2024-03-25T19:45:14.429 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:45:14.430179+0000 2024-03-25T19:45:14.429 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:14.429 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:14.430 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:45:14.430 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:45:14.430 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:45:14.430 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:14.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.429+0000 7f321c148640 1 Processor -- start 2024-03-25T19:45:14.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.429+0000 7f321c148640 1 -- start start 2024-03-25T19:45:14.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.429+0000 7f321c148640 10 monclient: init 2024-03-25T19:45:14.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.430+0000 7f321c148640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:14.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.430+0000 7f321c148640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:45:14.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.430+0000 7f321c148640 10 monclient(hunting): picked mon.noname-c con 0x7f32140b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T19:45:14.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.430+0000 7f321c148640 10 monclient(hunting): picked mon.noname-a con 0x7f32140b1730 addr v1:172.21.15.44:6789/0 2024-03-25T19:45:14.431 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.430+0000 7f321c148640 10 monclient(hunting): picked mon.noname-b con 0x7f321414be90 addr v1:172.21.15.44:6790/0 2024-03-25T19:45:14.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.430+0000 7f321c148640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32140af940 con 0x7f32140b1730 2024-03-25T19:45:14.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.430+0000 7f321c148640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32140b0a70 con 0x7f32140b4d60 2024-03-25T19:45:14.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.430+0000 7f321c148640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32140b0d50 con 0x7f321414be90 2024-03-25T19:45:14.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.430+0000 7f321c148640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:14.432 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.430+0000 7f321c148640 10 monclient(hunting): authenticate will time out at 5978.601606s 2024-03-25T19:45:14.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.438+0000 7f3219ee4640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f32140b4d60 0x7f321413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:53648/0 (socket says 172.21.15.44:53648) 2024-03-25T19:45:14.439 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.438+0000 7f3219ee4640 1 -- 172.21.15.44:0/3595510043 learned_addr learned my addr 172.21.15.44:0/3595510043 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:45:14.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.442+0000 7f3218ee2640 1 -- 172.21.15.44:0/3595510043 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2417566861 0 0) 0x7f32140b0a70 con 0x7f32140b4d60 2024-03-25T19:45:14.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.442+0000 7f3218ee2640 10 monclient(hunting): _negotiate 2024-03-25T19:45:14.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.442+0000 7f3218ee2640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:14.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.442+0000 7f3218ee2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:14.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.442+0000 7f3218ee2640 10 monclient(hunting): my global_id is 4663 2024-03-25T19:45:14.443 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.443+0000 7f3218ee2640 1 -- 172.21.15.44:0/3595510043 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3200003520 con 0x7f32140b4d60 2024-03-25T19:45:14.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.449+0000 7f3218ee2640 1 -- 172.21.15.44:0/3595510043 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3368455606 0 0) 0x7f3200003520 con 0x7f32140b4d60 2024-03-25T19:45:14.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.449+0000 7f3218ee2640 1 -- 172.21.15.44:0/3595510043 >> v1:172.21.15.44:6790/0 conn(0x7f321414be90 legacy=0x7f321414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:45:14.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.449+0000 7f3218ee2640 1 -- 172.21.15.44:0/3595510043 >> v1:172.21.15.44:6789/0 conn(0x7f32140b1730 legacy=0x7f32140b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:45:14.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.449+0000 7f3218ee2640 10 monclient: _finish_hunting 0 2024-03-25T19:45:14.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.449+0000 7f3218ee2640 1 monclient: found mon.noname-c 2024-03-25T19:45:14.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.449+0000 7f3218ee2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:14.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.449+0000 7f3218ee2640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:45:14.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.449+0000 7f3218ee2640 1 -- 172.21.15.44:0/3595510043 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32140b1030 con 0x7f32140b4d60 2024-03-25T19:45:14.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.449+0000 7f3218ee2640 10 monclient: _finish_auth 0 2024-03-25T19:45:14.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.449+0000 7f3218ee2640 10 monclient: _check_auth_tickets 2024-03-25T19:45:14.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.449+0000 7f3218ee2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:14.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.449+0000 7f3218ee2640 1 -- 172.21.15.44:0/3595510043 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f3210001690 con 0x7f32140b4d60 2024-03-25T19:45:14.450 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.449+0000 7f3218ee2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:14.451 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.449+0000 7f3218ee2640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:14.451 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.449+0000 7f3218ee2640 10 monclient: dump: 2024-03-25T19:45:14.451 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:14.451 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:14.451 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:14.451 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:14.451 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:14.451 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:14.451 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:14.451 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:14.451 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:14.451 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:14.451 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.449+0000 7f321c148640 5 monclient: authenticate success, global_id 4663 2024-03-25T19:45:14.451 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.449+0000 7f321c148640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:14.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.461+0000 7f3218ee2640 1 -- 172.21.15.44:0/3595510043 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3210001860 con 0x7f32140b4d60 2024-03-25T19:45:14.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.461+0000 7f3218ee2640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:14.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.461+0000 7f321c148640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:14.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.461+0000 7f321c148640 4 set_mon_vals no callback set 2024-03-25T19:45:14.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.462+0000 7f321c148640 10 monclient: shutdown 2024-03-25T19:45:14.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.462+0000 7f321c148640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:14.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.462+0000 7f321c148640 1 -- 172.21.15.44:0/3595510043 >> v1:172.21.15.111:6789/0 conn(0x7f32140b4d60 legacy=0x7f321413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:14.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.462+0000 7f321c148640 1 -- 172.21.15.44:0/3595510043 shutdown_connections 2024-03-25T19:45:14.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.462+0000 7f321c148640 1 -- 172.21.15.44:0/3595510043 >> 172.21.15.44:0/3595510043 conn(0x7f32140617a0 msgr2=0x7f3214061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:14.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.462+0000 7f321c148640 1 -- 172.21.15.44:0/3595510043 shutdown_connections 2024-03-25T19:45:14.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.462+0000 7f321c148640 1 -- 172.21.15.44:0/3595510043 wait complete. 2024-03-25T19:45:14.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.462+0000 7f321c148640 10 monclient: build_initial_monmap 2024-03-25T19:45:14.464 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.462+0000 7f321c148640 10 monclient: monmap: 2024-03-25T19:45:14.464 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:45:14.464 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:14.464 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:45:14.464 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:45:14.464 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:14.464 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:45:14.465 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:45:14.465 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:45:14.465 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:45:14.465 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:14.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.463+0000 7f321c148640 1 librados: starting msgr at 2024-03-25T19:45:14.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.463+0000 7f321c148640 1 librados: starting objecter 2024-03-25T19:45:14.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.463+0000 7f321c148640 1 Processor -- start 2024-03-25T19:45:14.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.463+0000 7f321c148640 1 -- start start 2024-03-25T19:45:14.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.463+0000 7f321c148640 1 librados: setting wanted keys 2024-03-25T19:45:14.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.463+0000 7f321c148640 1 librados: calling monclient init 2024-03-25T19:45:14.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.463+0000 7f321c148640 10 monclient: init 2024-03-25T19:45:14.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.463+0000 7f321c148640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:14.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.463+0000 7f321c148640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:45:14.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.463+0000 7f321c148640 10 monclient(hunting): picked mon.noname-b con 0x7f32140b1730 addr v1:172.21.15.111:6789/0 2024-03-25T19:45:14.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.463+0000 7f321c148640 10 monclient(hunting): picked mon.noname-c con 0x7f32140b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T19:45:14.465 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.463+0000 7f321c148640 10 monclient(hunting): picked mon.noname-a con 0x7f321414be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:45:14.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.464+0000 7f321c148640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3214140d30 con 0x7f321414be90 2024-03-25T19:45:14.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.464+0000 7f321c148640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32141386c0 con 0x7f32140b1730 2024-03-25T19:45:14.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.464+0000 7f321c148640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3214138850 con 0x7f32140b4d60 2024-03-25T19:45:14.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.464+0000 7f321c148640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:14.466 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.464+0000 7f321c148640 10 monclient(hunting): authenticate will time out at 5978.634954s 2024-03-25T19:45:14.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.473+0000 7f3219ee4640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f32140b1730 0x7f32141dab90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:53690/0 (socket says 172.21.15.44:53690) 2024-03-25T19:45:14.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.473+0000 7f3219ee4640 1 -- 172.21.15.44:0/2592561488 learned_addr learned my addr 172.21.15.44:0/2592561488 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:45:14.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.474+0000 7f320a7fc640 10 client.?.objecter ms_handle_connect 0x7f32140b1730 2024-03-25T19:45:14.474 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.474+0000 7f320a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:14.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.477+0000 7f320a7fc640 10 client.?.objecter ms_handle_connect 0x7f321414be90 2024-03-25T19:45:14.477 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.477+0000 7f320a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:14.486 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.486+0000 7f320a7fc640 10 client.?.objecter ms_handle_connect 0x7f32140b4d60 2024-03-25T19:45:14.486 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.486+0000 7f320a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:14.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.544+0000 7f320a7fc640 1 -- 172.21.15.44:0/2592561488 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242736595 0 0) 0x7f3214140d30 con 0x7f321414be90 2024-03-25T19:45:14.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.544+0000 7f320a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:14.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.544+0000 7f320a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:14.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.544+0000 7f320a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:14.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.544+0000 7f320a7fc640 10 monclient(hunting): my global_id is 4677 2024-03-25T19:45:14.545 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.544+0000 7f320a7fc640 1 -- 172.21.15.44:0/2592561488 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31f8004eb0 con 0x7f321414be90 2024-03-25T19:45:14.589 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.588+0000 7f320a7fc640 1 -- 172.21.15.44:0/2592561488 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1673156348 0 0) 0x7f31f8004eb0 con 0x7f321414be90 2024-03-25T19:45:14.589 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.588+0000 7f320a7fc640 1 -- 172.21.15.44:0/2592561488 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3214140d30 con 0x7f321414be90 2024-03-25T19:45:14.589 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.588+0000 7f320a7fc640 1 -- 172.21.15.44:0/2592561488 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f320c001b20 con 0x7f321414be90 2024-03-25T19:45:14.589 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.588+0000 7f320a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:14.589 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.588+0000 7f320a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:14.589 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.588+0000 7f320a7fc640 10 monclient(hunting): dump: 2024-03-25T19:45:14.589 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:14.589 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:14.590 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:14.590 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:14.590 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:14.590 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:14.590 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:14.590 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:14.590 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:14.590 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:14.590 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.588+0000 7f320a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:14.610 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.609+0000 7f320a7fc640 1 -- 172.21.15.44:0/2592561488 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 274309581 0 0) 0x7f3214140d30 con 0x7f321414be90 2024-03-25T19:45:14.610 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.609+0000 7f320a7fc640 1 -- 172.21.15.44:0/2592561488 >> v1:172.21.15.44:6790/0 conn(0x7f32140b4d60 legacy=0x7f3214131d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:14.610 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.609+0000 7f320a7fc640 1 -- 172.21.15.44:0/2592561488 >> v1:172.21.15.111:6789/0 conn(0x7f32140b1730 legacy=0x7f32141dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:14.610 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.609+0000 7f320a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:45:14.610 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.609+0000 7f320a7fc640 1 monclient: found mon.a 2024-03-25T19:45:14.610 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.609+0000 7f320a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:14.610 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.609+0000 7f320a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:45:14.610 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.609+0000 7f320a7fc640 1 -- 172.21.15.44:0/2592561488 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3214140380 con 0x7f321414be90 2024-03-25T19:45:14.610 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.609+0000 7f320a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:45:14.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.609+0000 7f320a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:45:14.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.609+0000 7f320a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:14.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.609+0000 7f321c148640 5 monclient: authenticate success, global_id 4677 2024-03-25T19:45:14.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.609+0000 7f321c148640 10 monclient: _renew_subs 2024-03-25T19:45:14.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.609+0000 7f321c148640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:45:14.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.609+0000 7f321c148640 1 -- 172.21.15.44:0/2592561488 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32141405d0 con 0x7f321414be90 2024-03-25T19:45:14.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.609+0000 7f321c148640 10 client.4677.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:14.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.609+0000 7f321c148640 10 monclient: _renew_subs 2024-03-25T19:45:14.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.609+0000 7f321c148640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:45:14.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.609+0000 7f321c148640 1 -- 172.21.15.44:0/2592561488 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3214137f70 con 0x7f321414be90 2024-03-25T19:45:14.611 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.609+0000 7f321c148640 1 librados: init done 2024-03-25T19:45:14.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.611+0000 7f321c148640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:14.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.611+0000 7f321c148640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:14.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.611+0000 7f321c148640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:45:14.612 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.611+0000 7f321c148640 1 -- 172.21.15.44:0/2592561488 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32140627b0 con 0x7f321414be90 2024-03-25T19:45:14.973 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.973+0000 7f320a7fc640 1 -- 172.21.15.44:0/2592561488 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f320c002070 con 0x7f321414be90 2024-03-25T19:45:14.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.973+0000 7f320a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:14.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.973+0000 7f320a7fc640 1 -- 172.21.15.44:0/2592561488 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f320c003970 con 0x7f321414be90 2024-03-25T19:45:14.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.973+0000 7f320a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:14.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.973+0000 7f320a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:14.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.973+0000 7f320a7fc640 10 monclient: dump: 2024-03-25T19:45:14.974 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:14.974 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:14.974 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:14.974 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:14.974 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:14.974 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:14.974 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:14.975 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:14.975 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:14.975 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:14.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.973+0000 7f320bfff640 4 set_mon_vals no callback set 2024-03-25T19:45:14.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.974+0000 7f320a7fc640 1 -- 172.21.15.44:0/2592561488 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f320c003b90 con 0x7f321414be90 2024-03-25T19:45:14.975 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.975+0000 7f3219ee4640 10 monclient: get_auth_request con 0x7f31f806dd90 auth_method 0 2024-03-25T19:45:14.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.975+0000 7f320a7fc640 10 client.4677.objecter ms_handle_connect 0x7f31f806dd90 2024-03-25T19:45:14.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.981+0000 7f320a7fc640 1 -- 172.21.15.44:0/2592561488 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f320c0535b0 con 0x7f321414be90 2024-03-25T19:45:14.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.981+0000 7f320a7fc640 10 client.4677.objecter ms_dispatch 0x7f3214000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:14.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.981+0000 7f320a7fc640 3 client.4677.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:14.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.981+0000 7f320a7fc640 3 client.4677.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:14.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.981+0000 7f320a7fc640 20 client.4677.objecter dump_active .. 0 homeless 2024-03-25T19:45:14.981 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.981+0000 7f320a7fc640 1 -- 172.21.15.44:0/2592561488 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f320c04f0c0 con 0x7f321414be90 2024-03-25T19:45:14.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.981+0000 7f320a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:14.982 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:14.981+0000 7f320a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:15.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:15.232+0000 7f321c148640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:15.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:15.232+0000 7f321c148640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:15.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:15.232+0000 7f321c148640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:45:15.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:15.232+0000 7f321c148640 1 -- 172.21.15.44:0/2592561488 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3214061e30 con 0x7f321414be90 2024-03-25T19:45:15.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:15.265+0000 7f320a7fc640 1 -- 172.21.15.44:0/2592561488 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f320c050ff0 con 0x7f321414be90 2024-03-25T19:45:15.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:15.265+0000 7f320a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:15.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:15.265+0000 7f320a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:15.266 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:45:15.266 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:15.268 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:15.267+0000 7f321c148640 10 librados: watch_flush enter 2024-03-25T19:45:15.268 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:15.267+0000 7f321c148640 10 librados: watch_flush exit 2024-03-25T19:45:15.268 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:15.267+0000 7f321c148640 20 client.4677.objecter shutdown clearing up homeless session... 2024-03-25T19:45:15.268 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:15.267+0000 7f321c148640 10 client.4677.objecter successfully canceled tick 2024-03-25T19:45:15.268 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:15.267+0000 7f321c148640 1 -- 172.21.15.44:0/2592561488 >> v1:172.21.15.111:6800/3883215793 conn(0x7f31f806dd90 legacy=0x7f31f8070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:15.268 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:15.267+0000 7f321c148640 10 monclient: shutdown 2024-03-25T19:45:15.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:15.267+0000 7f321c148640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:15.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:15.267+0000 7f321c148640 1 -- 172.21.15.44:0/2592561488 >> v1:172.21.15.44:6789/0 conn(0x7f321414be90 legacy=0x7f32141e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:15.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:15.267+0000 7f321c148640 1 -- 172.21.15.44:0/2592561488 shutdown_connections 2024-03-25T19:45:15.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:15.267+0000 7f321c148640 1 -- 172.21.15.44:0/2592561488 >> 172.21.15.44:0/2592561488 conn(0x7f32140617a0 msgr2=0x7f32140b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:15.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:15.268+0000 7f321c148640 1 -- 172.21.15.44:0/2592561488 shutdown_connections 2024-03-25T19:45:15.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:15.268+0000 7f321c148640 1 -- 172.21.15.44:0/2592561488 wait complete. 2024-03-25T19:45:15.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:15.268+0000 7f321c148640 1 librados: shutdown 2024-03-25T19:45:15.283 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:22.090 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:22.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.205+0000 7fd84c6e4640 10 monclient: get_monmap_and_config 2024-03-25T19:45:22.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.205+0000 7fd84c6e4640 10 monclient: build_initial_monmap 2024-03-25T19:45:22.206 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.205+0000 7fd84c6e4640 10 monclient: monmap: 2024-03-25T19:45:22.206 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:45:22.206 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:22.206 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:45:22.206985+0000 2024-03-25T19:45:22.206 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:45:22.206985+0000 2024-03-25T19:45:22.206 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:22.206 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:22.206 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:45:22.207 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:45:22.207 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:45:22.207 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:22.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.206+0000 7fd84c6e4640 1 Processor -- start 2024-03-25T19:45:22.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.206+0000 7fd84c6e4640 1 -- start start 2024-03-25T19:45:22.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.206+0000 7fd84c6e4640 10 monclient: init 2024-03-25T19:45:22.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.207+0000 7fd84c6e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:22.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.207+0000 7fd84c6e4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:45:22.208 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.207+0000 7fd84c6e4640 10 monclient(hunting): picked mon.noname-a con 0x7fd84406c380 addr v1:172.21.15.44:6789/0 2024-03-25T19:45:22.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.207+0000 7fd84c6e4640 10 monclient(hunting): picked mon.noname-b con 0x7fd8440b12d0 addr v1:172.21.15.44:6790/0 2024-03-25T19:45:22.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.207+0000 7fd84c6e4640 10 monclient(hunting): picked mon.noname-c con 0x7fd8440aedf0 addr v1:172.21.15.111:6789/0 2024-03-25T19:45:22.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.207+0000 7fd84c6e4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8440b16f0 con 0x7fd84406c380 2024-03-25T19:45:22.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.207+0000 7fd84c6e4640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8440adfe0 con 0x7fd8440aedf0 2024-03-25T19:45:22.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.207+0000 7fd84c6e4640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8440add00 con 0x7fd8440b12d0 2024-03-25T19:45:22.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.207+0000 7fd84c6e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:22.209 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.207+0000 7fd84c6e4640 10 monclient(hunting): authenticate will time out at 5986.378251s 2024-03-25T19:45:22.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.212+0000 7fd84ac81640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd8440aedf0 0x7fd84414e050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:60152/0 (socket says 172.21.15.44:60152) 2024-03-25T19:45:22.212 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.212+0000 7fd84ac81640 1 -- 172.21.15.44:0/614389678 learned_addr learned my addr 172.21.15.44:0/614389678 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:45:22.223 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.222+0000 7fd84947e640 1 -- 172.21.15.44:0/614389678 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1283370055 0 0) 0x7fd8440adfe0 con 0x7fd8440aedf0 2024-03-25T19:45:22.223 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.222+0000 7fd84947e640 10 monclient(hunting): _negotiate 2024-03-25T19:45:22.223 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.222+0000 7fd84947e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:22.223 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.222+0000 7fd84947e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:22.223 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.222+0000 7fd84947e640 10 monclient(hunting): my global_id is 4669 2024-03-25T19:45:22.223 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.222+0000 7fd84947e640 1 -- 172.21.15.44:0/614389678 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd830003520 con 0x7fd8440aedf0 2024-03-25T19:45:22.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.229+0000 7fd84947e640 1 -- 172.21.15.44:0/614389678 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 725074757 0 0) 0x7fd830003520 con 0x7fd8440aedf0 2024-03-25T19:45:22.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.229+0000 7fd84947e640 1 -- 172.21.15.44:0/614389678 >> v1:172.21.15.44:6790/0 conn(0x7fd8440b12d0 legacy=0x7fd8440ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:45:22.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.229+0000 7fd84947e640 1 -- 172.21.15.44:0/614389678 >> v1:172.21.15.44:6789/0 conn(0x7fd84406c380 legacy=0x7fd8440b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:22.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.229+0000 7fd84947e640 10 monclient: _finish_hunting 0 2024-03-25T19:45:22.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.229+0000 7fd84947e640 1 monclient: found mon.noname-c 2024-03-25T19:45:22.230 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.229+0000 7fd84947e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:22.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.229+0000 7fd84947e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:45:22.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.229+0000 7fd84947e640 1 -- 172.21.15.44:0/614389678 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8440ae2c0 con 0x7fd8440aedf0 2024-03-25T19:45:22.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.229+0000 7fd84947e640 10 monclient: _finish_auth 0 2024-03-25T19:45:22.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.229+0000 7fd84947e640 10 monclient: _check_auth_tickets 2024-03-25T19:45:22.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.229+0000 7fd84947e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:22.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.229+0000 7fd84947e640 1 -- 172.21.15.44:0/614389678 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fd83c001a10 con 0x7fd8440aedf0 2024-03-25T19:45:22.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.229+0000 7fd84947e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:22.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.229+0000 7fd84947e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:22.232 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.229+0000 7fd84947e640 10 monclient: dump: 2024-03-25T19:45:22.232 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:22.232 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:22.232 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:22.232 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:22.232 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:22.232 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:22.233 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:22.233 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:22.233 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:22.233 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:22.233 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.229+0000 7fd84c6e4640 5 monclient: authenticate success, global_id 4669 2024-03-25T19:45:22.233 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.229+0000 7fd84c6e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:22.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.235+0000 7fd84947e640 1 -- 172.21.15.44:0/614389678 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd83c001b70 con 0x7fd8440aedf0 2024-03-25T19:45:22.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.235+0000 7fd84947e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:22.235 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.235+0000 7fd84947e640 1 -- 172.21.15.44:0/614389678 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fd83c001fc0 con 0x7fd8440aedf0 2024-03-25T19:45:22.236 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.235+0000 7fd84c6e4640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:22.236 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.235+0000 7fd84c6e4640 4 set_mon_vals no callback set 2024-03-25T19:45:22.236 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.235+0000 7fd84947e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:22.236 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.235+0000 7fd84947e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:22.236 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.235+0000 7fd84947e640 10 monclient: dump: 2024-03-25T19:45:22.236 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:22.236 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:22.236 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:22.236 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:22.236 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:22.236 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:22.236 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:22.237 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:22.237 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:22.237 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:22.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.236+0000 7fd84c6e4640 10 monclient: shutdown 2024-03-25T19:45:22.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.236+0000 7fd84c6e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:22.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.236+0000 7fd84c6e4640 1 -- 172.21.15.44:0/614389678 >> v1:172.21.15.111:6789/0 conn(0x7fd8440aedf0 legacy=0x7fd84414e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:22.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.236+0000 7fd84c6e4640 1 -- 172.21.15.44:0/614389678 shutdown_connections 2024-03-25T19:45:22.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.236+0000 7fd84c6e4640 1 -- 172.21.15.44:0/614389678 >> 172.21.15.44:0/614389678 conn(0x7fd8440b5d10 msgr2=0x7fd8440b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:22.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.236+0000 7fd84c6e4640 1 -- 172.21.15.44:0/614389678 shutdown_connections 2024-03-25T19:45:22.239 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.236+0000 7fd84c6e4640 1 -- 172.21.15.44:0/614389678 wait complete. 2024-03-25T19:45:22.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.236+0000 7fd84c6e4640 10 monclient: build_initial_monmap 2024-03-25T19:45:22.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.236+0000 7fd84c6e4640 10 monclient: monmap: 2024-03-25T19:45:22.240 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:45:22.240 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:22.240 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:45:22.240 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:45:22.240 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:22.240 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:45:22.240 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:45:22.240 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:45:22.240 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:45:22.240 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:22.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.237+0000 7fd84c6e4640 1 librados: starting msgr at 2024-03-25T19:45:22.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.237+0000 7fd84c6e4640 1 librados: starting objecter 2024-03-25T19:45:22.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.237+0000 7fd84c6e4640 1 Processor -- start 2024-03-25T19:45:22.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.237+0000 7fd84c6e4640 1 -- start start 2024-03-25T19:45:22.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.237+0000 7fd84c6e4640 1 librados: setting wanted keys 2024-03-25T19:45:22.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.237+0000 7fd84c6e4640 1 librados: calling monclient init 2024-03-25T19:45:22.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.237+0000 7fd84c6e4640 10 monclient: init 2024-03-25T19:45:22.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.238+0000 7fd84c6e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:22.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.238+0000 7fd84c6e4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:22.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.238+0000 7fd84c6e4640 10 monclient(hunting): picked mon.noname-a con 0x7fd84406c380 addr v1:172.21.15.44:6789/0 2024-03-25T19:45:22.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.238+0000 7fd84c6e4640 10 monclient(hunting): picked mon.noname-b con 0x7fd8440aedf0 addr v1:172.21.15.111:6789/0 2024-03-25T19:45:22.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.238+0000 7fd84c6e4640 10 monclient(hunting): picked mon.noname-c con 0x7fd8440b12d0 addr v1:172.21.15.44:6790/0 2024-03-25T19:45:22.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.238+0000 7fd84c6e4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd84406c8e0 con 0x7fd84406c380 2024-03-25T19:45:22.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.238+0000 7fd84c6e4640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8441cacf0 con 0x7fd8440aedf0 2024-03-25T19:45:22.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.238+0000 7fd84c6e4640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8441c3bb0 con 0x7fd8440b12d0 2024-03-25T19:45:22.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.238+0000 7fd84c6e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:22.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.238+0000 7fd84c6e4640 10 monclient(hunting): authenticate will time out at 5986.409123s 2024-03-25T19:45:22.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.244+0000 7fd849c7f640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd8440aedf0 0x7fd8441d7d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:60178/0 (socket says 172.21.15.44:60178) 2024-03-25T19:45:22.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.244+0000 7fd849c7f640 1 -- 172.21.15.44:0/1665638850 learned_addr learned my addr 172.21.15.44:0/1665638850 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:45:22.246 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.246+0000 7fd83affd640 10 client.?.objecter ms_handle_connect 0x7fd8440aedf0 2024-03-25T19:45:22.246 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.246+0000 7fd83affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:22.254 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.254+0000 7fd83affd640 10 client.?.objecter ms_handle_connect 0x7fd8440b12d0 2024-03-25T19:45:22.254 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.254+0000 7fd83affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:22.255 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.255+0000 7fd83affd640 1 -- 172.21.15.44:0/1665638850 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940432873 0 0) 0x7fd8441cacf0 con 0x7fd8440aedf0 2024-03-25T19:45:22.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.255+0000 7fd83affd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:22.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.255+0000 7fd83affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:22.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.255+0000 7fd83affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:22.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.255+0000 7fd83affd640 10 monclient(hunting): my global_id is 4672 2024-03-25T19:45:22.256 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.255+0000 7fd83affd640 1 -- 172.21.15.44:0/1665638850 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd828004eb0 con 0x7fd8440aedf0 2024-03-25T19:45:22.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.263+0000 7fd83affd640 10 client.?.objecter ms_handle_connect 0x7fd84406c380 2024-03-25T19:45:22.263 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.263+0000 7fd83affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:22.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.265+0000 7fd83affd640 1 -- 172.21.15.44:0/1665638850 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 140396581 0 0) 0x7fd8441c3bb0 con 0x7fd8440b12d0 2024-03-25T19:45:22.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.265+0000 7fd83affd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:22.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.265+0000 7fd83affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:22.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.265+0000 7fd83affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:22.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.265+0000 7fd83affd640 10 monclient(hunting): my global_id is 4661 2024-03-25T19:45:22.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.265+0000 7fd83affd640 1 -- 172.21.15.44:0/1665638850 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8441cacf0 con 0x7fd8440b12d0 2024-03-25T19:45:22.267 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.267+0000 7fd83affd640 1 -- 172.21.15.44:0/1665638850 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 504899203 0 0) 0x7fd828004eb0 con 0x7fd8440aedf0 2024-03-25T19:45:22.267 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.267+0000 7fd83affd640 1 -- 172.21.15.44:0/1665638850 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8441c3bb0 con 0x7fd8440aedf0 2024-03-25T19:45:22.268 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.267+0000 7fd83affd640 1 -- 172.21.15.44:0/1665638850 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fd8340017d0 con 0x7fd8440aedf0 2024-03-25T19:45:22.268 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.267+0000 7fd83affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:22.268 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.267+0000 7fd83affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:22.268 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.267+0000 7fd83affd640 10 monclient(hunting): dump: 2024-03-25T19:45:22.268 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:22.268 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:22.268 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:22.268 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:22.268 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:22.268 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:22.268 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:22.268 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:22.268 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:22.269 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:22.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.267+0000 7fd83affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:22.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.273+0000 7fd83affd640 1 -- 172.21.15.44:0/1665638850 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3195613722 0 0) 0x7fd84406c8e0 con 0x7fd84406c380 2024-03-25T19:45:22.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.273+0000 7fd83affd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:22.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.273+0000 7fd83affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:22.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.273+0000 7fd83affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:22.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.273+0000 7fd83affd640 10 monclient(hunting): my global_id is 4680 2024-03-25T19:45:22.273 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.273+0000 7fd83affd640 1 -- 172.21.15.44:0/1665638850 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd828004eb0 con 0x7fd84406c380 2024-03-25T19:45:22.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.274+0000 7fd83affd640 1 -- 172.21.15.44:0/1665638850 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 659599168 0 0) 0x7fd8441cacf0 con 0x7fd8440b12d0 2024-03-25T19:45:22.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.274+0000 7fd83affd640 1 -- 172.21.15.44:0/1665638850 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd84406c8e0 con 0x7fd8440b12d0 2024-03-25T19:45:22.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.274+0000 7fd83affd640 1 -- 172.21.15.44:0/1665638850 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fd83c001ce0 con 0x7fd8440b12d0 2024-03-25T19:45:22.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.274+0000 7fd83affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:22.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.274+0000 7fd83affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:22.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.274+0000 7fd83affd640 10 monclient(hunting): dump: 2024-03-25T19:45:22.275 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:22.275 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:22.275 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:22.275 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:22.275 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:22.275 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:22.275 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:22.275 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:22.275 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:22.275 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:22.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.279+0000 7fd83affd640 1 -- 172.21.15.44:0/1665638850 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2908217582 0 0) 0x7fd8441c3bb0 con 0x7fd8440aedf0 2024-03-25T19:45:22.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.279+0000 7fd83affd640 1 -- 172.21.15.44:0/1665638850 >> v1:172.21.15.44:6790/0 conn(0x7fd8440b12d0 legacy=0x7fd8441e23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:22.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.279+0000 7fd83affd640 1 -- 172.21.15.44:0/1665638850 >> v1:172.21.15.44:6789/0 conn(0x7fd84406c380 legacy=0x7fd8441c34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:22.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.279+0000 7fd83affd640 10 monclient: _finish_hunting 0 2024-03-25T19:45:22.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.279+0000 7fd83affd640 1 monclient: found mon.b 2024-03-25T19:45:22.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.279+0000 7fd83affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:22.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.279+0000 7fd83affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:45:22.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.279+0000 7fd83affd640 1 -- 172.21.15.44:0/1665638850 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8441c18b0 con 0x7fd8440aedf0 2024-03-25T19:45:22.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.279+0000 7fd83affd640 10 monclient: _finish_auth 0 2024-03-25T19:45:22.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.279+0000 7fd83affd640 10 monclient: _check_auth_tickets 2024-03-25T19:45:22.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.279+0000 7fd83affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:22.280 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.279+0000 7fd84c6e4640 5 monclient: authenticate success, global_id 4672 2024-03-25T19:45:22.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.279+0000 7fd84c6e4640 10 monclient: _renew_subs 2024-03-25T19:45:22.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.279+0000 7fd84c6e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:45:22.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.279+0000 7fd84c6e4640 1 -- 172.21.15.44:0/1665638850 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8441c2a90 con 0x7fd8440aedf0 2024-03-25T19:45:22.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.279+0000 7fd84c6e4640 10 client.4672.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:22.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.279+0000 7fd84c6e4640 10 monclient: _renew_subs 2024-03-25T19:45:22.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.279+0000 7fd84c6e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:45:22.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.279+0000 7fd84c6e4640 1 -- 172.21.15.44:0/1665638850 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8441d8600 con 0x7fd8440aedf0 2024-03-25T19:45:22.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.279+0000 7fd84c6e4640 1 librados: init done 2024-03-25T19:45:22.282 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.280+0000 7fd84c6e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:22.282 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.280+0000 7fd84c6e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:22.282 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.280+0000 7fd84c6e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:45:22.282 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.280+0000 7fd84c6e4640 1 -- 172.21.15.44:0/1665638850 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd84406d790 con 0x7fd8440aedf0 2024-03-25T19:45:22.550 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.550+0000 7fd83affd640 1 -- 172.21.15.44:0/1665638850 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd834001ce0 con 0x7fd8440aedf0 2024-03-25T19:45:22.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.550+0000 7fd83affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:22.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.550+0000 7fd83affd640 1 -- 172.21.15.44:0/1665638850 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fd834004860 con 0x7fd8440aedf0 2024-03-25T19:45:22.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.550+0000 7fd83affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:22.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.550+0000 7fd83affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:22.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.550+0000 7fd83affd640 10 monclient: dump: 2024-03-25T19:45:22.551 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:22.551 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:22.551 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:22.551 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:22.551 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:22.551 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:22.552 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:22.552 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:22.552 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:22.552 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:22.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.550+0000 7fd848c7d640 4 set_mon_vals no callback set 2024-03-25T19:45:22.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.551+0000 7fd83affd640 1 -- 172.21.15.44:0/1665638850 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7fd83401a830 con 0x7fd8440aedf0 2024-03-25T19:45:22.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.552+0000 7fd84a480640 10 monclient: get_auth_request con 0x7fd82806df70 auth_method 0 2024-03-25T19:45:22.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.552+0000 7fd83affd640 10 client.4672.objecter ms_handle_connect 0x7fd82806df70 2024-03-25T19:45:22.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.556+0000 7fd83affd640 1 -- 172.21.15.44:0/1665638850 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7fd834054560 con 0x7fd8440aedf0 2024-03-25T19:45:22.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.556+0000 7fd83affd640 10 client.4672.objecter ms_dispatch 0x7fd844000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:22.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.556+0000 7fd83affd640 3 client.4672.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:22.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.556+0000 7fd83affd640 3 client.4672.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:22.556 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.556+0000 7fd83affd640 20 client.4672.objecter dump_active .. 0 homeless 2024-03-25T19:45:22.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.558+0000 7fd83affd640 1 -- 172.21.15.44:0/1665638850 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd83404fea0 con 0x7fd8440aedf0 2024-03-25T19:45:22.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.559+0000 7fd83affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:22.559 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.559+0000 7fd83affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:22.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.791+0000 7fd84c6e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:22.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.791+0000 7fd84c6e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:22.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.791+0000 7fd84c6e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:45:22.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.791+0000 7fd84c6e4640 1 -- 172.21.15.44:0/1665638850 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8441d7a30 con 0x7fd8440aedf0 2024-03-25T19:45:22.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.813+0000 7fd83affd640 1 -- 172.21.15.44:0/1665638850 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7fd8441d7a30 con 0x7fd8440aedf0 2024-03-25T19:45:22.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.813+0000 7fd83affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:22.813 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.813+0000 7fd83affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:22.813 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:45:22.814 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:22.817 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.815+0000 7fd84c6e4640 10 librados: watch_flush enter 2024-03-25T19:45:22.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.815+0000 7fd84c6e4640 10 librados: watch_flush exit 2024-03-25T19:45:22.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.815+0000 7fd84c6e4640 20 client.4672.objecter shutdown clearing up homeless session... 2024-03-25T19:45:22.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.815+0000 7fd84c6e4640 10 client.4672.objecter successfully canceled tick 2024-03-25T19:45:22.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.815+0000 7fd84c6e4640 1 -- 172.21.15.44:0/1665638850 >> v1:172.21.15.111:6800/3883215793 conn(0x7fd82806df70 legacy=0x7fd828070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:22.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.815+0000 7fd84c6e4640 10 monclient: shutdown 2024-03-25T19:45:22.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.815+0000 7fd84c6e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:22.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.815+0000 7fd84c6e4640 1 -- 172.21.15.44:0/1665638850 >> v1:172.21.15.111:6789/0 conn(0x7fd8440aedf0 legacy=0x7fd8441d7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:22.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.815+0000 7fd84c6e4640 1 -- 172.21.15.44:0/1665638850 shutdown_connections 2024-03-25T19:45:22.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.815+0000 7fd84c6e4640 1 -- 172.21.15.44:0/1665638850 >> 172.21.15.44:0/1665638850 conn(0x7fd8440b5d10 msgr2=0x7fd8441cd420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:22.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.815+0000 7fd84c6e4640 1 -- 172.21.15.44:0/1665638850 shutdown_connections 2024-03-25T19:45:22.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.815+0000 7fd84c6e4640 1 -- 172.21.15.44:0/1665638850 wait complete. 2024-03-25T19:45:22.818 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:22.815+0000 7fd84c6e4640 1 librados: shutdown 2024-03-25T19:45:22.830 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:25.632 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:10:16.533 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:25.632 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:10:16.511 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:25.633 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:10:16.513 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:25.639 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:10:16.541 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:25.659 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:10:16.538 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:25.667 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:10:16.568 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:25.707 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:10:16.609 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:25.715 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:10:16.617 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:25.749 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:10:16.650 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:29.636 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:29.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.753+0000 7f695a3a3640 10 monclient: get_monmap_and_config 2024-03-25T19:45:29.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.753+0000 7f695a3a3640 10 monclient: build_initial_monmap 2024-03-25T19:45:29.752 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.753+0000 7f695a3a3640 10 monclient: monmap: 2024-03-25T19:45:29.752 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:45:29.753 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:29.753 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:45:29.753405+0000 2024-03-25T19:45:29.753 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:45:29.753405+0000 2024-03-25T19:45:29.753 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:29.753 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:29.753 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:45:29.753 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:45:29.753 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:45:29.753 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:29.754 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.754+0000 7f695a3a3640 1 Processor -- start 2024-03-25T19:45:29.754 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.754+0000 7f695a3a3640 1 -- start start 2024-03-25T19:45:29.754 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.754+0000 7f695a3a3640 10 monclient: init 2024-03-25T19:45:29.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.755+0000 7f695a3a3640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:29.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.755+0000 7f695a3a3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:45:29.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.755+0000 7f695a3a3640 10 monclient(hunting): picked mon.noname-c con 0x7f69540a96e0 addr v1:172.21.15.111:6789/0 2024-03-25T19:45:29.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.755+0000 7f695a3a3640 10 monclient(hunting): picked mon.noname-b con 0x7f69540b1b10 addr v1:172.21.15.44:6790/0 2024-03-25T19:45:29.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.755+0000 7f695a3a3640 10 monclient(hunting): picked mon.noname-a con 0x7f6954146060 addr v1:172.21.15.44:6789/0 2024-03-25T19:45:29.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.755+0000 7f695a3a3640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6954140040 con 0x7f6954146060 2024-03-25T19:45:29.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.755+0000 7f695a3a3640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69540b1fa0 con 0x7f69540a96e0 2024-03-25T19:45:29.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.755+0000 7f695a3a3640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69540b24c0 con 0x7f69540b1b10 2024-03-25T19:45:29.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.755+0000 7f695a3a3640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:29.755 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.755+0000 7f695a3a3640 10 monclient(hunting): authenticate will time out at 5993.925057s 2024-03-25T19:45:29.758 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.759+0000 7f6958940640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f6954146060 0x7f695414dcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:57758/0 (socket says 172.21.15.44:57758) 2024-03-25T19:45:29.758 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.759+0000 7f6958940640 1 -- 172.21.15.44:0/199148109 learned_addr learned my addr 172.21.15.44:0/199148109 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:45:29.781 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.780+0000 7f6952ffd640 1 -- 172.21.15.44:0/199148109 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562285647 0 0) 0x7f69540b1fa0 con 0x7f69540a96e0 2024-03-25T19:45:29.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.780+0000 7f6952ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:29.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.780+0000 7f6952ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:29.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.780+0000 7f6952ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:29.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.780+0000 7f6952ffd640 10 monclient(hunting): my global_id is 4675 2024-03-25T19:45:29.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.780+0000 7f6952ffd640 1 -- 172.21.15.44:0/199148109 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f693c003520 con 0x7f69540a96e0 2024-03-25T19:45:29.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.781+0000 7f6952ffd640 1 -- 172.21.15.44:0/199148109 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3388176321 0 0) 0x7f69540b24c0 con 0x7f69540b1b10 2024-03-25T19:45:29.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.781+0000 7f6952ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:29.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.781+0000 7f6952ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:29.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.781+0000 7f6952ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:29.782 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.781+0000 7f6952ffd640 10 monclient(hunting): my global_id is 4664 2024-03-25T19:45:29.783 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.781+0000 7f6952ffd640 1 -- 172.21.15.44:0/199148109 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69540b1fa0 con 0x7f69540b1b10 2024-03-25T19:45:29.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.795+0000 7f6952ffd640 1 -- 172.21.15.44:0/199148109 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3080360427 0 0) 0x7f69540b1fa0 con 0x7f69540b1b10 2024-03-25T19:45:29.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.795+0000 7f6952ffd640 1 -- 172.21.15.44:0/199148109 >> v1:172.21.15.111:6789/0 conn(0x7f69540a96e0 legacy=0x7f69540b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:29.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.795+0000 7f6952ffd640 1 -- 172.21.15.44:0/199148109 >> v1:172.21.15.44:6789/0 conn(0x7f6954146060 legacy=0x7f695414dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:29.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.795+0000 7f6952ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:45:29.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.795+0000 7f6952ffd640 1 monclient: found mon.noname-b 2024-03-25T19:45:29.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.795+0000 7f6952ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:29.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.795+0000 7f6952ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:45:29.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.795+0000 7f6952ffd640 1 -- 172.21.15.44:0/199148109 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69540b15b0 con 0x7f69540b1b10 2024-03-25T19:45:29.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.795+0000 7f6952ffd640 10 monclient: _finish_auth 0 2024-03-25T19:45:29.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.795+0000 7f6952ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:45:29.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.795+0000 7f6952ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:29.795 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.795+0000 7f6952ffd640 1 -- 172.21.15.44:0/199148109 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f69480016e0 con 0x7f69540b1b10 2024-03-25T19:45:29.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.795+0000 7f695a3a3640 5 monclient: authenticate success, global_id 4664 2024-03-25T19:45:29.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.795+0000 7f695a3a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:29.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.795+0000 7f6952ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:29.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.795+0000 7f6952ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:29.796 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.795+0000 7f6952ffd640 10 monclient: dump: 2024-03-25T19:45:29.796 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:29.796 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:29.796 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:29.796 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:29.796 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:29.796 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:29.796 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:29.796 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:29.797 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:29.797 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:29.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.795+0000 7f695a3a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:29.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.805+0000 7f6952ffd640 1 -- 172.21.15.44:0/199148109 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6948001840 con 0x7f69540b1b10 2024-03-25T19:45:29.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.805+0000 7f6952ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:29.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.805+0000 7f6952ffd640 1 -- 172.21.15.44:0/199148109 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6948001c10 con 0x7f69540b1b10 2024-03-25T19:45:29.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.805+0000 7f695a3a3640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:29.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.805+0000 7f695a3a3640 4 set_mon_vals no callback set 2024-03-25T19:45:29.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.805+0000 7f6952ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:29.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.805+0000 7f6952ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:29.804 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.805+0000 7f6952ffd640 10 monclient: dump: 2024-03-25T19:45:29.804 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:29.804 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:29.804 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:29.804 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:29.805 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:29.805 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:29.805 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:29.805 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:29.805 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:29.805 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:29.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.805+0000 7f695a3a3640 10 monclient: shutdown 2024-03-25T19:45:29.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.805+0000 7f695a3a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:29.806 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.805+0000 7f695a3a3640 1 -- 172.21.15.44:0/199148109 >> v1:172.21.15.44:6790/0 conn(0x7f69540b1b10 legacy=0x7f6954145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:29.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.805+0000 7f695a3a3640 1 -- 172.21.15.44:0/199148109 shutdown_connections 2024-03-25T19:45:29.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.805+0000 7f695a3a3640 1 -- 172.21.15.44:0/199148109 >> 172.21.15.44:0/199148109 conn(0x7f69540b7ea0 msgr2=0x7f69540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:29.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.805+0000 7f695a3a3640 1 -- 172.21.15.44:0/199148109 shutdown_connections 2024-03-25T19:45:29.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.805+0000 7f695a3a3640 1 -- 172.21.15.44:0/199148109 wait complete. 2024-03-25T19:45:29.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.806+0000 7f695a3a3640 10 monclient: build_initial_monmap 2024-03-25T19:45:29.807 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.806+0000 7f695a3a3640 10 monclient: monmap: 2024-03-25T19:45:29.807 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:45:29.807 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:29.807 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:45:29.807 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:45:29.807 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:29.807 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:45:29.807 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:45:29.808 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:45:29.808 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:45:29.808 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:29.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.807+0000 7f695a3a3640 1 librados: starting msgr at 2024-03-25T19:45:29.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.807+0000 7f695a3a3640 1 librados: starting objecter 2024-03-25T19:45:29.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.807+0000 7f695a3a3640 1 Processor -- start 2024-03-25T19:45:29.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.807+0000 7f695a3a3640 1 -- start start 2024-03-25T19:45:29.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.807+0000 7f695a3a3640 1 librados: setting wanted keys 2024-03-25T19:45:29.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.807+0000 7f695a3a3640 1 librados: calling monclient init 2024-03-25T19:45:29.808 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.807+0000 7f695a3a3640 10 monclient: init 2024-03-25T19:45:29.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.808+0000 7f695a3a3640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:29.809 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.808+0000 7f695a3a3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:45:29.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.808+0000 7f695a3a3640 10 monclient(hunting): picked mon.noname-b con 0x7f69540a96e0 addr v1:172.21.15.111:6789/0 2024-03-25T19:45:29.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.808+0000 7f695a3a3640 10 monclient(hunting): picked mon.noname-c con 0x7f6954146060 addr v1:172.21.15.44:6790/0 2024-03-25T19:45:29.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.808+0000 7f695a3a3640 10 monclient(hunting): picked mon.noname-a con 0x7f695413ae50 addr v1:172.21.15.44:6789/0 2024-03-25T19:45:29.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.808+0000 7f695a3a3640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69540ad950 con 0x7f695413ae50 2024-03-25T19:45:29.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.808+0000 7f695a3a3640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69540aea20 con 0x7f69540a96e0 2024-03-25T19:45:29.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.808+0000 7f695a3a3640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69540ac6a0 con 0x7f6954146060 2024-03-25T19:45:29.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.808+0000 7f695a3a3640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:29.810 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.808+0000 7f695a3a3640 10 monclient(hunting): authenticate will time out at 5993.978187s 2024-03-25T19:45:29.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.825+0000 7f69537fe640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f6954146060 0x7f69540af1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:58658/0 (socket says 172.21.15.44:58658) 2024-03-25T19:45:29.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.825+0000 7f69537fe640 1 -- 172.21.15.44:0/1190277083 learned_addr learned my addr 172.21.15.44:0/1190277083 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:45:29.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.825+0000 7f6950ff9640 10 client.?.objecter ms_handle_connect 0x7f6954146060 2024-03-25T19:45:29.824 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.825+0000 7f6950ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:29.831 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.832+0000 7f6950ff9640 10 client.?.objecter ms_handle_connect 0x7f695413ae50 2024-03-25T19:45:29.831 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.832+0000 7f6950ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:29.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.838+0000 7f6950ff9640 1 -- 172.21.15.44:0/1190277083 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280724841 0 0) 0x7f69540ac6a0 con 0x7f6954146060 2024-03-25T19:45:29.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.838+0000 7f6950ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:29.837 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.838+0000 7f6950ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:29.838 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.838+0000 7f6950ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:29.838 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.838+0000 7f6950ff9640 10 monclient(hunting): my global_id is 4667 2024-03-25T19:45:29.838 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.838+0000 7f6950ff9640 1 -- 172.21.15.44:0/1190277083 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6940004eb0 con 0x7f6954146060 2024-03-25T19:45:29.845 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.846+0000 7f6950ff9640 1 -- 172.21.15.44:0/1190277083 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693420369 0 0) 0x7f69540ad950 con 0x7f695413ae50 2024-03-25T19:45:29.845 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.846+0000 7f6950ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:29.845 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.846+0000 7f6950ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:29.845 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.846+0000 7f6950ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:29.845 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.846+0000 7f6950ff9640 10 monclient(hunting): my global_id is 4686 2024-03-25T19:45:29.845 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.846+0000 7f6950ff9640 1 -- 172.21.15.44:0/1190277083 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69540ac6a0 con 0x7f695413ae50 2024-03-25T19:45:29.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.847+0000 7f6950ff9640 10 client.?.objecter ms_handle_connect 0x7f69540a96e0 2024-03-25T19:45:29.846 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.847+0000 7f6950ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:29.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.853+0000 7f6950ff9640 1 -- 172.21.15.44:0/1190277083 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2279906959 0 0) 0x7f6940004eb0 con 0x7f6954146060 2024-03-25T19:45:29.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.853+0000 7f6950ff9640 1 -- 172.21.15.44:0/1190277083 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69540ad950 con 0x7f6954146060 2024-03-25T19:45:29.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.853+0000 7f6950ff9640 1 -- 172.21.15.44:0/1190277083 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f69480016e0 con 0x7f6954146060 2024-03-25T19:45:29.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.853+0000 7f6950ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:29.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.854+0000 7f6950ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:29.853 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.854+0000 7f6950ff9640 10 monclient(hunting): dump: 2024-03-25T19:45:29.853 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:29.853 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:29.853 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:29.854 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:29.854 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:29.854 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:29.854 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:29.854 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:29.854 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:29.854 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:29.854 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.854+0000 7f6950ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:29.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.856+0000 7f6950ff9640 1 -- 172.21.15.44:0/1190277083 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2447219666 0 0) 0x7f69540aea20 con 0x7f69540a96e0 2024-03-25T19:45:29.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.856+0000 7f6950ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:29.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.856+0000 7f6950ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:29.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.856+0000 7f6950ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:29.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.856+0000 7f6950ff9640 10 monclient(hunting): my global_id is 4678 2024-03-25T19:45:29.855 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.856+0000 7f6950ff9640 1 -- 172.21.15.44:0/1190277083 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6940004eb0 con 0x7f69540a96e0 2024-03-25T19:45:29.859 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.859+0000 7f6950ff9640 1 -- 172.21.15.44:0/1190277083 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 450470700 0 0) 0x7f69540ac6a0 con 0x7f695413ae50 2024-03-25T19:45:29.859 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.859+0000 7f6950ff9640 1 -- 172.21.15.44:0/1190277083 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69540aea20 con 0x7f695413ae50 2024-03-25T19:45:29.859 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.859+0000 7f6950ff9640 1 -- 172.21.15.44:0/1190277083 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f694c001b00 con 0x7f695413ae50 2024-03-25T19:45:29.859 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.859+0000 7f6950ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:29.859 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.859+0000 7f6950ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:29.859 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.859+0000 7f6950ff9640 10 monclient(hunting): dump: 2024-03-25T19:45:29.859 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:29.859 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:29.859 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:29.860 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:29.860 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:29.860 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:29.860 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:29.860 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:29.860 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:29.860 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:29.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.869+0000 7f6950ff9640 1 -- 172.21.15.44:0/1190277083 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2801115496 0 0) 0x7f69540aea20 con 0x7f695413ae50 2024-03-25T19:45:29.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.869+0000 7f6950ff9640 1 -- 172.21.15.44:0/1190277083 >> v1:172.21.15.44:6790/0 conn(0x7f6954146060 legacy=0x7f69540af1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:29.869 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.869+0000 7f6950ff9640 1 -- 172.21.15.44:0/1190277083 >> v1:172.21.15.111:6789/0 conn(0x7f69540a96e0 legacy=0x7f69540ad110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:29.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.869+0000 7f6950ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:45:29.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.869+0000 7f6950ff9640 1 monclient: found mon.a 2024-03-25T19:45:29.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.869+0000 7f6950ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:29.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.869+0000 7f6950ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:45:29.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.869+0000 7f6950ff9640 1 -- 172.21.15.44:0/1190277083 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69541e5ed0 con 0x7f695413ae50 2024-03-25T19:45:29.870 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.869+0000 7f6950ff9640 10 monclient: _finish_auth 0 2024-03-25T19:45:29.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.869+0000 7f6950ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:45:29.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.869+0000 7f6950ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:29.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.869+0000 7f695a3a3640 5 monclient: authenticate success, global_id 4686 2024-03-25T19:45:29.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.869+0000 7f695a3a3640 10 monclient: _renew_subs 2024-03-25T19:45:29.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.869+0000 7f695a3a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:45:29.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.869+0000 7f695a3a3640 1 -- 172.21.15.44:0/1190277083 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69541e2ea0 con 0x7f695413ae50 2024-03-25T19:45:29.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.870+0000 7f695a3a3640 10 client.4686.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:29.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.870+0000 7f695a3a3640 10 monclient: _renew_subs 2024-03-25T19:45:29.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.870+0000 7f695a3a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:45:29.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.870+0000 7f695a3a3640 1 -- 172.21.15.44:0/1190277083 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69541e3380 con 0x7f695413ae50 2024-03-25T19:45:29.871 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.870+0000 7f695a3a3640 1 librados: init done 2024-03-25T19:45:29.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.871+0000 7f695a3a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:29.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.871+0000 7f695a3a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:29.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.871+0000 7f695a3a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:45:29.872 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:29.871+0000 7f695a3a3640 1 -- 172.21.15.44:0/1190277083 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69540adb00 con 0x7f695413ae50 2024-03-25T19:45:30.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.190+0000 7f6950ff9640 1 -- 172.21.15.44:0/1190277083 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f694c001f30 con 0x7f695413ae50 2024-03-25T19:45:30.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.190+0000 7f6950ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:30.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.190+0000 7f6950ff9640 1 -- 172.21.15.44:0/1190277083 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f694c004d00 con 0x7f695413ae50 2024-03-25T19:45:30.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.191+0000 7f6950ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:30.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.191+0000 7f6950ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:30.190 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.191+0000 7f6950ff9640 10 monclient: dump: 2024-03-25T19:45:30.191 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:30.191 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:30.191 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:30.191 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:30.191 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:30.191 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:30.191 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:30.191 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:30.191 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:30.191 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:30.191 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.191+0000 7f69527fc640 4 set_mon_vals no callback set 2024-03-25T19:45:30.192 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.192+0000 7f6950ff9640 1 -- 172.21.15.44:0/1190277083 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f694c004f20 con 0x7f695413ae50 2024-03-25T19:45:30.192 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.192+0000 7f6953fff640 10 monclient: get_auth_request con 0x7f694006df50 auth_method 0 2024-03-25T19:45:30.192 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.193+0000 7f6950ff9640 10 client.4686.objecter ms_handle_connect 0x7f694006df50 2024-03-25T19:45:30.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.199+0000 7f6950ff9640 1 -- 172.21.15.44:0/1190277083 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3794696201 0 0) 0x7f694c0549f0 con 0x7f695413ae50 2024-03-25T19:45:30.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.199+0000 7f6950ff9640 10 client.4686.objecter ms_dispatch 0x7f6954000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:30.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.199+0000 7f6950ff9640 3 client.4686.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:30.198 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.199+0000 7f6950ff9640 3 client.4686.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:30.199 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.199+0000 7f6950ff9640 20 client.4686.objecter dump_active .. 0 homeless 2024-03-25T19:45:30.199 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.199+0000 7f6950ff9640 1 -- 172.21.15.44:0/1190277083 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f694c050500 con 0x7f695413ae50 2024-03-25T19:45:30.199 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.199+0000 7f6950ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:30.199 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.199+0000 7f6950ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:30.446 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.447+0000 7f695a3a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:30.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.447+0000 7f695a3a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:30.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.447+0000 7f695a3a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:45:30.447 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.447+0000 7f695a3a3640 1 -- 172.21.15.44:0/1190277083 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69540a9ac0 con 0x7f695413ae50 2024-03-25T19:45:30.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.487+0000 7f6950ff9640 1 -- 172.21.15.44:0/1190277083 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1726162951) 0x7f694c052430 con 0x7f695413ae50 2024-03-25T19:45:30.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.487+0000 7f6950ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:30.487 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.487+0000 7f6950ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:30.487 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:45:30.487 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":10,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:35:31.752165+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:35:31.752165+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:30.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.490+0000 7f695a3a3640 10 librados: watch_flush enter 2024-03-25T19:45:30.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.490+0000 7f695a3a3640 10 librados: watch_flush exit 2024-03-25T19:45:30.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.490+0000 7f695a3a3640 20 client.4686.objecter shutdown clearing up homeless session... 2024-03-25T19:45:30.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.490+0000 7f695a3a3640 10 client.4686.objecter successfully canceled tick 2024-03-25T19:45:30.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.490+0000 7f695a3a3640 1 -- 172.21.15.44:0/1190277083 >> v1:172.21.15.111:6800/3883215793 conn(0x7f694006df50 legacy=0x7f6940070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:30.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.490+0000 7f695a3a3640 10 monclient: shutdown 2024-03-25T19:45:30.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.490+0000 7f695a3a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:30.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.490+0000 7f695a3a3640 1 -- 172.21.15.44:0/1190277083 >> v1:172.21.15.44:6789/0 conn(0x7f695413ae50 legacy=0x7f69540af8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:30.490 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.490+0000 7f695a3a3640 1 -- 172.21.15.44:0/1190277083 shutdown_connections 2024-03-25T19:45:30.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.490+0000 7f695a3a3640 1 -- 172.21.15.44:0/1190277083 >> 172.21.15.44:0/1190277083 conn(0x7f69540b7ea0 msgr2=0x7f695413de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:30.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.490+0000 7f695a3a3640 1 -- 172.21.15.44:0/1190277083 shutdown_connections 2024-03-25T19:45:30.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.490+0000 7f695a3a3640 1 -- 172.21.15.44:0/1190277083 wait complete. 2024-03-25T19:45:30.491 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:30.491+0000 7f695a3a3640 1 librados: shutdown 2024-03-25T19:45:30.502 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:37.309 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:37.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.425+0000 7ff56a82e640 10 monclient: get_monmap_and_config 2024-03-25T19:45:37.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.425+0000 7ff56a82e640 10 monclient: build_initial_monmap 2024-03-25T19:45:37.425 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.425+0000 7ff56a82e640 10 monclient: monmap: 2024-03-25T19:45:37.425 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:45:37.425 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:37.425 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:45:37.426195+0000 2024-03-25T19:45:37.426 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:45:37.426195+0000 2024-03-25T19:45:37.426 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:37.426 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:37.426 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:45:37.426 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:45:37.426 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:45:37.426 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:37.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.425+0000 7ff56a82e640 1 Processor -- start 2024-03-25T19:45:37.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.425+0000 7ff56a82e640 1 -- start start 2024-03-25T19:45:37.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.425+0000 7ff56a82e640 10 monclient: init 2024-03-25T19:45:37.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.426+0000 7ff56a82e640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:37.427 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.426+0000 7ff56a82e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:45:37.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.426+0000 7ff56a82e640 10 monclient(hunting): picked mon.noname-c con 0x7ff5640a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:45:37.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.426+0000 7ff56a82e640 10 monclient(hunting): picked mon.noname-b con 0x7ff56413f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:45:37.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.426+0000 7ff56a82e640 10 monclient(hunting): picked mon.noname-a con 0x7ff5641430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:45:37.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.426+0000 7ff56a82e640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff564142520 con 0x7ff5641430c0 2024-03-25T19:45:37.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.426+0000 7ff56a82e640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5640b28a0 con 0x7ff5640a98f0 2024-03-25T19:45:37.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.426+0000 7ff56a82e640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5640b2a30 con 0x7ff56413f710 2024-03-25T19:45:37.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.426+0000 7ff56a82e640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:37.428 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.426+0000 7ff56a82e640 10 monclient(hunting): authenticate will time out at 6001.597366s 2024-03-25T19:45:37.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.438+0000 7ff5637fe640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7ff56413f710 0x7ff5641429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:35230/0 (socket says 172.21.15.44:35230) 2024-03-25T19:45:37.438 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.438+0000 7ff5637fe640 1 -- 172.21.15.44:0/3277022549 learned_addr learned my addr 172.21.15.44:0/3277022549 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:45:37.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.452+0000 7ff562ffd640 1 -- 172.21.15.44:0/3277022549 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782747939 0 0) 0x7ff564142520 con 0x7ff5641430c0 2024-03-25T19:45:37.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.453+0000 7ff562ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:37.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.453+0000 7ff562ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:37.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.453+0000 7ff562ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:37.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.453+0000 7ff562ffd640 10 monclient(hunting): my global_id is 4689 2024-03-25T19:45:37.453 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.453+0000 7ff562ffd640 1 -- 172.21.15.44:0/3277022549 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff54c003520 con 0x7ff5641430c0 2024-03-25T19:45:37.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.458+0000 7ff562ffd640 1 -- 172.21.15.44:0/3277022549 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235569416 0 0) 0x7ff5640b2a30 con 0x7ff56413f710 2024-03-25T19:45:37.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.458+0000 7ff562ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:37.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.458+0000 7ff562ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:37.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.458+0000 7ff562ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:37.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.458+0000 7ff562ffd640 10 monclient(hunting): my global_id is 4670 2024-03-25T19:45:37.458 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.458+0000 7ff562ffd640 1 -- 172.21.15.44:0/3277022549 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff564142520 con 0x7ff56413f710 2024-03-25T19:45:37.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.458+0000 7ff562ffd640 1 -- 172.21.15.44:0/3277022549 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3377314541 0 0) 0x7ff54c003520 con 0x7ff5641430c0 2024-03-25T19:45:37.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.458+0000 7ff562ffd640 1 -- 172.21.15.44:0/3277022549 >> v1:172.21.15.44:6790/0 conn(0x7ff56413f710 legacy=0x7ff5641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:37.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.458+0000 7ff562ffd640 1 -- 172.21.15.44:0/3277022549 >> v1:172.21.15.111:6789/0 conn(0x7ff5640a98f0 legacy=0x7ff5640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:37.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.458+0000 7ff562ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:45:37.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.458+0000 7ff562ffd640 1 monclient: found mon.noname-a 2024-03-25T19:45:37.459 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.458+0000 7ff562ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:37.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.458+0000 7ff562ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:45:37.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.458+0000 7ff562ffd640 1 -- 172.21.15.44:0/3277022549 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5640b44c0 con 0x7ff5641430c0 2024-03-25T19:45:37.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.458+0000 7ff562ffd640 10 monclient: _finish_auth 0 2024-03-25T19:45:37.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.458+0000 7ff562ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:45:37.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.458+0000 7ff562ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:37.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.458+0000 7ff562ffd640 1 -- 172.21.15.44:0/3277022549 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff55c001a10 con 0x7ff5641430c0 2024-03-25T19:45:37.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.458+0000 7ff56a82e640 5 monclient: authenticate success, global_id 4689 2024-03-25T19:45:37.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.458+0000 7ff56a82e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:37.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.458+0000 7ff562ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:37.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.458+0000 7ff562ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:37.460 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.458+0000 7ff562ffd640 10 monclient: dump: 2024-03-25T19:45:37.460 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:37.460 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:37.460 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:37.460 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:37.460 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:37.461 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:37.461 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:37.461 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:37.461 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:37.461 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:37.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.458+0000 7ff56a82e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:37.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.466+0000 7ff562ffd640 1 -- 172.21.15.44:0/3277022549 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff55c001b70 con 0x7ff5641430c0 2024-03-25T19:45:37.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.466+0000 7ff562ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:37.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.466+0000 7ff562ffd640 1 -- 172.21.15.44:0/3277022549 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff55c001f80 con 0x7ff5641430c0 2024-03-25T19:45:37.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.466+0000 7ff562ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:37.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.467+0000 7ff562ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:37.467 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.467+0000 7ff562ffd640 10 monclient: dump: 2024-03-25T19:45:37.467 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:37.467 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:37.467 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:37.467 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:37.467 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:37.468 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:37.468 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:37.468 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:37.468 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:37.468 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:37.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.467+0000 7ff56a82e640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:37.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.467+0000 7ff56a82e640 4 set_mon_vals no callback set 2024-03-25T19:45:37.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.467+0000 7ff56a82e640 10 monclient: shutdown 2024-03-25T19:45:37.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.467+0000 7ff56a82e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:37.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.467+0000 7ff56a82e640 1 -- 172.21.15.44:0/3277022549 >> v1:172.21.15.44:6789/0 conn(0x7ff5641430c0 legacy=0x7ff56414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:37.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.468+0000 7ff56a82e640 1 -- 172.21.15.44:0/3277022549 shutdown_connections 2024-03-25T19:45:37.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.468+0000 7ff56a82e640 1 -- 172.21.15.44:0/3277022549 >> 172.21.15.44:0/3277022549 conn(0x7ff5640b7ea0 msgr2=0x7ff5640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:37.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.468+0000 7ff56a82e640 1 -- 172.21.15.44:0/3277022549 shutdown_connections 2024-03-25T19:45:37.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.468+0000 7ff56a82e640 1 -- 172.21.15.44:0/3277022549 wait complete. 2024-03-25T19:45:37.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.468+0000 7ff56a82e640 10 monclient: build_initial_monmap 2024-03-25T19:45:37.470 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.469+0000 7ff56a82e640 10 monclient: monmap: 2024-03-25T19:45:37.470 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:45:37.470 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:37.470 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:45:37.470 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:45:37.470 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:37.470 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:45:37.470 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:45:37.470 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:45:37.470 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:45:37.471 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:37.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.469+0000 7ff56a82e640 1 librados: starting msgr at 2024-03-25T19:45:37.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.469+0000 7ff56a82e640 1 librados: starting objecter 2024-03-25T19:45:37.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.469+0000 7ff56a82e640 1 Processor -- start 2024-03-25T19:45:37.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.469+0000 7ff56a82e640 1 -- start start 2024-03-25T19:45:37.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.469+0000 7ff56a82e640 1 librados: setting wanted keys 2024-03-25T19:45:37.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.469+0000 7ff56a82e640 1 librados: calling monclient init 2024-03-25T19:45:37.471 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.469+0000 7ff56a82e640 10 monclient: init 2024-03-25T19:45:37.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.470+0000 7ff56a82e640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:37.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.470+0000 7ff56a82e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:45:37.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.470+0000 7ff56a82e640 10 monclient(hunting): picked mon.noname-c con 0x7ff5640a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T19:45:37.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.470+0000 7ff56a82e640 10 monclient(hunting): picked mon.noname-a con 0x7ff56413f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:45:37.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.470+0000 7ff56a82e640 10 monclient(hunting): picked mon.noname-b con 0x7ff5641430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:45:37.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.470+0000 7ff56a82e640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff56413b410 con 0x7ff56413f710 2024-03-25T19:45:37.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.470+0000 7ff56a82e640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff56413b010 con 0x7ff5641430c0 2024-03-25T19:45:37.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.470+0000 7ff56a82e640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff564135230 con 0x7ff5640a98f0 2024-03-25T19:45:37.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.470+0000 7ff56a82e640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:37.472 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.470+0000 7ff56a82e640 10 monclient(hunting): authenticate will time out at 6001.641126s 2024-03-25T19:45:37.475 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.475+0000 7ff563fff640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7ff5640a98f0 0x7ff564130ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:35262/0 (socket says 172.21.15.44:35262) 2024-03-25T19:45:37.475 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.475+0000 7ff563fff640 1 -- 172.21.15.44:0/107015174 learned_addr learned my addr 172.21.15.44:0/107015174 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:45:37.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.476+0000 7ff560ff9640 10 client.?.objecter ms_handle_connect 0x7ff5640a98f0 2024-03-25T19:45:37.476 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.476+0000 7ff560ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:37.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.479+0000 7ff560ff9640 10 client.?.objecter ms_handle_connect 0x7ff5641430c0 2024-03-25T19:45:37.480 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.479+0000 7ff560ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:37.483 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.483+0000 7ff560ff9640 1 -- 172.21.15.44:0/107015174 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3634464402 0 0) 0x7ff56413b010 con 0x7ff5641430c0 2024-03-25T19:45:37.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.483+0000 7ff560ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:37.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.483+0000 7ff560ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:37.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.483+0000 7ff560ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:37.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.483+0000 7ff560ff9640 10 monclient(hunting): my global_id is 4681 2024-03-25T19:45:37.484 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.483+0000 7ff560ff9640 1 -- 172.21.15.44:0/107015174 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff550004eb0 con 0x7ff5641430c0 2024-03-25T19:45:37.492 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.492+0000 7ff560ff9640 1 -- 172.21.15.44:0/107015174 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 576157303 0 0) 0x7ff564135230 con 0x7ff5640a98f0 2024-03-25T19:45:37.492 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.492+0000 7ff560ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:37.492 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.492+0000 7ff560ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:37.492 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.492+0000 7ff560ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:37.492 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.492+0000 7ff560ff9640 10 monclient(hunting): my global_id is 4673 2024-03-25T19:45:37.492 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.492+0000 7ff560ff9640 1 -- 172.21.15.44:0/107015174 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff56413b010 con 0x7ff5640a98f0 2024-03-25T19:45:37.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.495+0000 7ff560ff9640 1 -- 172.21.15.44:0/107015174 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2623885600 0 0) 0x7ff550004eb0 con 0x7ff5641430c0 2024-03-25T19:45:37.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.495+0000 7ff560ff9640 1 -- 172.21.15.44:0/107015174 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff564135230 con 0x7ff5641430c0 2024-03-25T19:45:37.495 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.495+0000 7ff560ff9640 1 -- 172.21.15.44:0/107015174 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff55c001ec0 con 0x7ff5641430c0 2024-03-25T19:45:37.496 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.495+0000 7ff560ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:37.496 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.495+0000 7ff560ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:37.496 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.495+0000 7ff560ff9640 10 monclient(hunting): dump: 2024-03-25T19:45:37.496 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:37.496 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:37.496 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:37.496 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:37.496 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:37.496 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:37.496 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:37.496 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:37.496 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:37.496 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:37.496 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.495+0000 7ff560ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:37.502 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.502+0000 7ff560ff9640 1 -- 172.21.15.44:0/107015174 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3970251380 0 0) 0x7ff56413b010 con 0x7ff5640a98f0 2024-03-25T19:45:37.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.502+0000 7ff560ff9640 1 -- 172.21.15.44:0/107015174 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff550004eb0 con 0x7ff5640a98f0 2024-03-25T19:45:37.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.502+0000 7ff560ff9640 1 -- 172.21.15.44:0/107015174 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff554001990 con 0x7ff5640a98f0 2024-03-25T19:45:37.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.502+0000 7ff560ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:37.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.502+0000 7ff560ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:37.503 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.502+0000 7ff560ff9640 10 monclient(hunting): dump: 2024-03-25T19:45:37.503 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:37.503 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:37.503 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:37.503 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:37.503 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:37.503 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:37.503 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:37.504 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:37.504 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:37.504 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:37.509 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.508+0000 7ff560ff9640 1 -- 172.21.15.44:0/107015174 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 942028518 0 0) 0x7ff564135230 con 0x7ff5641430c0 2024-03-25T19:45:37.509 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.508+0000 7ff560ff9640 1 -- 172.21.15.44:0/107015174 >> v1:172.21.15.44:6790/0 conn(0x7ff5640a98f0 legacy=0x7ff564130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:37.509 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.508+0000 7ff560ff9640 1 -- 172.21.15.44:0/107015174 >> v1:172.21.15.44:6789/0 conn(0x7ff56413f710 legacy=0x7ff5641dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:37.509 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.508+0000 7ff560ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:45:37.509 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.508+0000 7ff560ff9640 1 monclient: found mon.b 2024-03-25T19:45:37.509 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.508+0000 7ff560ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:37.509 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.508+0000 7ff560ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:45:37.509 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.508+0000 7ff560ff9640 1 -- 172.21.15.44:0/107015174 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5641353c0 con 0x7ff5641430c0 2024-03-25T19:45:37.509 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.508+0000 7ff560ff9640 10 monclient: _finish_auth 0 2024-03-25T19:45:37.509 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.508+0000 7ff560ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:45:37.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.508+0000 7ff560ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:37.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.508+0000 7ff56a82e640 5 monclient: authenticate success, global_id 4681 2024-03-25T19:45:37.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.508+0000 7ff56a82e640 10 monclient: _renew_subs 2024-03-25T19:45:37.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.508+0000 7ff56a82e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:45:37.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.508+0000 7ff56a82e640 1 -- 172.21.15.44:0/107015174 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5640b78e0 con 0x7ff5641430c0 2024-03-25T19:45:37.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.508+0000 7ff56a82e640 10 client.4681.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:37.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.508+0000 7ff56a82e640 10 monclient: _renew_subs 2024-03-25T19:45:37.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.508+0000 7ff56a82e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:45:37.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.508+0000 7ff56a82e640 1 -- 172.21.15.44:0/107015174 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5641317d0 con 0x7ff5641430c0 2024-03-25T19:45:37.510 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.508+0000 7ff56a82e640 1 librados: init done 2024-03-25T19:45:37.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.510+0000 7ff56a82e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:37.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.510+0000 7ff56a82e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:37.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.510+0000 7ff56a82e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:45:37.511 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.510+0000 7ff56a82e640 1 -- 172.21.15.44:0/107015174 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5640b7c80 con 0x7ff5641430c0 2024-03-25T19:45:37.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.791+0000 7ff560ff9640 1 -- 172.21.15.44:0/107015174 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff55c0020e0 con 0x7ff5641430c0 2024-03-25T19:45:37.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.791+0000 7ff560ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:37.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.791+0000 7ff560ff9640 1 -- 172.21.15.44:0/107015174 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7ff55c003dc0 con 0x7ff5641430c0 2024-03-25T19:45:37.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.791+0000 7ff560ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:37.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.791+0000 7ff560ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:37.791 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.791+0000 7ff560ff9640 10 monclient: dump: 2024-03-25T19:45:37.791 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:37.792 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:37.792 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:37.792 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:37.792 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:37.792 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:37.792 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:37.792 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:37.792 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:37.792 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:37.792 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.791+0000 7ff5627fc640 4 set_mon_vals no callback set 2024-03-25T19:45:37.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.792+0000 7ff560ff9640 1 -- 172.21.15.44:0/107015174 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7ff55c019d90 con 0x7ff5641430c0 2024-03-25T19:45:37.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.792+0000 7ff563fff640 10 monclient: get_auth_request con 0x7ff55006de30 auth_method 0 2024-03-25T19:45:37.793 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.793+0000 7ff560ff9640 10 client.4681.objecter ms_handle_connect 0x7ff55006de30 2024-03-25T19:45:37.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.797+0000 7ff560ff9640 1 -- 172.21.15.44:0/107015174 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1275227418 0 0) 0x7ff55c053940 con 0x7ff5641430c0 2024-03-25T19:45:37.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.797+0000 7ff560ff9640 10 client.4681.objecter ms_dispatch 0x7ff564000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:45:37.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.797+0000 7ff560ff9640 3 client.4681.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:45:37.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.797+0000 7ff560ff9640 3 client.4681.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:45:37.797 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.797+0000 7ff560ff9640 20 client.4681.objecter dump_active .. 0 homeless 2024-03-25T19:45:37.798 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.797+0000 7ff560ff9640 1 -- 172.21.15.44:0/107015174 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff55c04f450 con 0x7ff5641430c0 2024-03-25T19:45:37.798 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.797+0000 7ff560ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:37.798 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:37.797+0000 7ff560ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:38.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:38.050+0000 7ff56a82e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:38.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:38.050+0000 7ff56a82e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:38.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:38.050+0000 7ff56a82e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:45:38.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:38.050+0000 7ff56a82e640 1 -- 172.21.15.44:0/107015174 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff56413b5c0 con 0x7ff5641430c0 2024-03-25T19:45:38.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:38.073+0000 7ff560ff9640 1 -- 172.21.15.44:0/107015174 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2782837909) 0x7ff56413b5c0 con 0x7ff5641430c0 2024-03-25T19:45:38.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:38.073+0000 7ff560ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:38.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:38.073+0000 7ff560ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:38.074 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:45:38.074 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":11,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:45:30.548832+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:45:30.548832+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:38.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:38.076+0000 7ff56a82e640 10 librados: watch_flush enter 2024-03-25T19:45:38.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:38.076+0000 7ff56a82e640 10 librados: watch_flush exit 2024-03-25T19:45:38.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:38.076+0000 7ff56a82e640 20 client.4681.objecter shutdown clearing up homeless session... 2024-03-25T19:45:38.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:38.076+0000 7ff56a82e640 10 client.4681.objecter successfully canceled tick 2024-03-25T19:45:38.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:38.076+0000 7ff56a82e640 1 -- 172.21.15.44:0/107015174 >> v1:172.21.15.111:6800/3883215793 conn(0x7ff55006de30 legacy=0x7ff5500702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:38.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:38.076+0000 7ff56a82e640 10 monclient: shutdown 2024-03-25T19:45:38.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:38.076+0000 7ff56a82e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:38.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:38.076+0000 7ff56a82e640 1 -- 172.21.15.44:0/107015174 >> v1:172.21.15.111:6789/0 conn(0x7ff5641430c0 legacy=0x7ff5641db2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:38.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:38.076+0000 7ff56a82e640 1 -- 172.21.15.44:0/107015174 shutdown_connections 2024-03-25T19:45:38.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:38.076+0000 7ff56a82e640 1 -- 172.21.15.44:0/107015174 >> 172.21.15.44:0/107015174 conn(0x7ff5640b7ea0 msgr2=0x7ff564074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:38.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:38.076+0000 7ff56a82e640 1 -- 172.21.15.44:0/107015174 shutdown_connections 2024-03-25T19:45:38.077 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:38.076+0000 7ff56a82e640 1 -- 172.21.15.44:0/107015174 wait complete. 2024-03-25T19:45:38.078 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:38.076+0000 7ff56a82e640 1 librados: shutdown 2024-03-25T19:45:38.088 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:44.895 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:45.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.013+0000 7f4550d13640 10 monclient: get_monmap_and_config 2024-03-25T19:45:45.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.013+0000 7f4550d13640 10 monclient: build_initial_monmap 2024-03-25T19:45:45.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.013+0000 7f4550d13640 10 monclient: monmap: 2024-03-25T19:45:45.014 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:45:45.014 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:45.014 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:45:45.014417+0000 2024-03-25T19:45:45.014 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:45:45.014417+0000 2024-03-25T19:45:45.014 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:45.014 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:45.014 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:45:45.014 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:45:45.015 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:45:45.015 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:45.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.013+0000 7f4550d13640 1 Processor -- start 2024-03-25T19:45:45.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.014+0000 7f4550d13640 1 -- start start 2024-03-25T19:45:45.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.014+0000 7f4550d13640 10 monclient: init 2024-03-25T19:45:45.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.015+0000 7f4550d13640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:45.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.015+0000 7f4550d13640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:45.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.015+0000 7f4550d13640 10 monclient(hunting): picked mon.noname-b con 0x7f454c0a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T19:45:45.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.015+0000 7f4550d13640 10 monclient(hunting): picked mon.noname-c con 0x7f454c13f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:45:45.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.015+0000 7f4550d13640 10 monclient(hunting): picked mon.noname-a con 0x7f454c1430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:45:45.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.015+0000 7f4550d13640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454c142520 con 0x7f454c1430c0 2024-03-25T19:45:45.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.015+0000 7f4550d13640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454c0b28a0 con 0x7f454c13f710 2024-03-25T19:45:45.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.015+0000 7f4550d13640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454c0b2a30 con 0x7f454c0a98f0 2024-03-25T19:45:45.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.015+0000 7f4550d13640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:45.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.015+0000 7f4550d13640 10 monclient(hunting): authenticate will time out at 6009.186056s 2024-03-25T19:45:45.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.025+0000 7f4549d9b640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f454c13f710 0x7f454c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:39454/0 (socket says 172.21.15.44:39454) 2024-03-25T19:45:45.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.025+0000 7f4549d9b640 1 -- 172.21.15.44:0/3542875959 learned_addr learned my addr 172.21.15.44:0/3542875959 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:45:45.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.039+0000 7f454959a640 1 -- 172.21.15.44:0/3542875959 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2738976610 0 0) 0x7f454c0b28a0 con 0x7f454c13f710 2024-03-25T19:45:45.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.039+0000 7f454959a640 10 monclient(hunting): _negotiate 2024-03-25T19:45:45.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.039+0000 7f454959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:45.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.039+0000 7f454959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:45.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.039+0000 7f454959a640 10 monclient(hunting): my global_id is 4684 2024-03-25T19:45:45.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.039+0000 7f454959a640 1 -- 172.21.15.44:0/3542875959 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4534003520 con 0x7f454c13f710 2024-03-25T19:45:45.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.039+0000 7f454959a640 1 -- 172.21.15.44:0/3542875959 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683463656 0 0) 0x7f454c0b2a30 con 0x7f454c0a98f0 2024-03-25T19:45:45.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.039+0000 7f454959a640 10 monclient(hunting): _negotiate 2024-03-25T19:45:45.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.039+0000 7f454959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:45.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.039+0000 7f454959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:45.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.039+0000 7f454959a640 10 monclient(hunting): my global_id is 4676 2024-03-25T19:45:45.040 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.039+0000 7f454959a640 1 -- 172.21.15.44:0/3542875959 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f454c0b28a0 con 0x7f454c0a98f0 2024-03-25T19:45:45.047 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.047+0000 7f454959a640 1 -- 172.21.15.44:0/3542875959 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3084345455 0 0) 0x7f454c0b28a0 con 0x7f454c0a98f0 2024-03-25T19:45:45.048 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.047+0000 7f454959a640 1 -- 172.21.15.44:0/3542875959 >> v1:172.21.15.111:6789/0 conn(0x7f454c13f710 legacy=0x7f454c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:45.048 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.047+0000 7f454959a640 1 -- 172.21.15.44:0/3542875959 >> v1:172.21.15.44:6789/0 conn(0x7f454c1430c0 legacy=0x7f454c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:45.048 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.047+0000 7f454959a640 10 monclient: _finish_hunting 0 2024-03-25T19:45:45.048 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.047+0000 7f454959a640 1 monclient: found mon.noname-b 2024-03-25T19:45:45.048 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.047+0000 7f454959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:45.048 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.047+0000 7f454959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:45:45.048 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.047+0000 7f454959a640 1 -- 172.21.15.44:0/3542875959 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f454c0b44c0 con 0x7f454c0a98f0 2024-03-25T19:45:45.048 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.047+0000 7f454959a640 10 monclient: _finish_auth 0 2024-03-25T19:45:45.048 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.047+0000 7f454959a640 10 monclient: _check_auth_tickets 2024-03-25T19:45:45.048 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.047+0000 7f454959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:45.048 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.047+0000 7f454959a640 1 -- 172.21.15.44:0/3542875959 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f453c001670 con 0x7f454c0a98f0 2024-03-25T19:45:45.048 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.047+0000 7f454959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:45.048 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.047+0000 7f454959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:45.048 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.047+0000 7f454959a640 10 monclient: dump: 2024-03-25T19:45:45.049 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:45.049 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:45.049 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:45.049 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:45.049 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:45.049 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:45.049 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:45.049 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:45.049 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:45.049 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:45.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.047+0000 7f4550d13640 5 monclient: authenticate success, global_id 4676 2024-03-25T19:45:45.049 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.047+0000 7f4550d13640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:45.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.057+0000 7f454959a640 1 -- 172.21.15.44:0/3542875959 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f453c0017d0 con 0x7f454c0a98f0 2024-03-25T19:45:45.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.057+0000 7f454959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:45.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.057+0000 7f454959a640 1 -- 172.21.15.44:0/3542875959 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f453c001ba0 con 0x7f454c0a98f0 2024-03-25T19:45:45.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.057+0000 7f454959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:45.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.057+0000 7f454959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:45.057 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.057+0000 7f454959a640 10 monclient: dump: 2024-03-25T19:45:45.057 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:45.057 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:45.057 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:45.057 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:45.057 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:45.057 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:45.058 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:45.058 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:45.058 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:45.058 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:45.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.057+0000 7f4550d13640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:45.058 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.057+0000 7f4550d13640 4 set_mon_vals no callback set 2024-03-25T19:45:45.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.057+0000 7f4550d13640 10 monclient: shutdown 2024-03-25T19:45:45.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.057+0000 7f4550d13640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:45.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.057+0000 7f4550d13640 1 -- 172.21.15.44:0/3542875959 >> v1:172.21.15.44:6790/0 conn(0x7f454c0a98f0 legacy=0x7f454c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:45.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.057+0000 7f4550d13640 1 -- 172.21.15.44:0/3542875959 shutdown_connections 2024-03-25T19:45:45.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.057+0000 7f4550d13640 1 -- 172.21.15.44:0/3542875959 >> 172.21.15.44:0/3542875959 conn(0x7f454c0b7ea0 msgr2=0x7f454c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:45.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.058+0000 7f4550d13640 1 -- 172.21.15.44:0/3542875959 shutdown_connections 2024-03-25T19:45:45.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.058+0000 7f4550d13640 1 -- 172.21.15.44:0/3542875959 wait complete. 2024-03-25T19:45:45.060 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.058+0000 7f4550d13640 10 monclient: build_initial_monmap 2024-03-25T19:45:45.061 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.058+0000 7f4550d13640 10 monclient: monmap: 2024-03-25T19:45:45.061 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:45:45.061 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:45.061 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:45:45.061 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:45:45.061 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:45.061 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:45:45.061 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:45:45.061 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:45:45.061 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:45:45.061 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:45.061 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.059+0000 7f4550d13640 1 librados: starting msgr at 2024-03-25T19:45:45.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.059+0000 7f4550d13640 1 librados: starting objecter 2024-03-25T19:45:45.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.059+0000 7f4550d13640 1 Processor -- start 2024-03-25T19:45:45.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.059+0000 7f4550d13640 1 -- start start 2024-03-25T19:45:45.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.059+0000 7f4550d13640 1 librados: setting wanted keys 2024-03-25T19:45:45.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.059+0000 7f4550d13640 1 librados: calling monclient init 2024-03-25T19:45:45.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.059+0000 7f4550d13640 10 monclient: init 2024-03-25T19:45:45.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.060+0000 7f4550d13640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:45.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.060+0000 7f4550d13640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:45.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.060+0000 7f4550d13640 10 monclient(hunting): picked mon.noname-a con 0x7f454c0a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:45:45.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.060+0000 7f4550d13640 10 monclient(hunting): picked mon.noname-b con 0x7f454c13f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:45:45.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.060+0000 7f4550d13640 10 monclient(hunting): picked mon.noname-c con 0x7f454c1430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:45:45.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.060+0000 7f4550d13640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454c13b180 con 0x7f454c0a98f0 2024-03-25T19:45:45.062 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.060+0000 7f4550d13640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454c12b950 con 0x7f454c13f710 2024-03-25T19:45:45.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.060+0000 7f4550d13640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454c0b7c00 con 0x7f454c1430c0 2024-03-25T19:45:45.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.060+0000 7f4550d13640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:45.063 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.060+0000 7f4550d13640 10 monclient(hunting): authenticate will time out at 6009.231036s 2024-03-25T19:45:45.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.074+0000 7f4549d9b640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f454c13f710 0x7f454c12ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:39480/0 (socket says 172.21.15.44:39480) 2024-03-25T19:45:45.074 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.074+0000 7f4549d9b640 1 -- 172.21.15.44:0/2642299030 learned_addr learned my addr 172.21.15.44:0/2642299030 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:45:45.080 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.080+0000 7f453affd640 10 client.?.objecter ms_handle_connect 0x7f454c0a98f0 2024-03-25T19:45:45.080 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.080+0000 7f453affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:45.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.083+0000 7f453affd640 10 client.?.objecter ms_handle_connect 0x7f454c1430c0 2024-03-25T19:45:45.084 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.083+0000 7f453affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:45.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.085+0000 7f453affd640 10 client.?.objecter ms_handle_connect 0x7f454c13f710 2024-03-25T19:45:45.085 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.085+0000 7f453affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:45.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.101+0000 7f453affd640 1 -- 172.21.15.44:0/2642299030 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1607629493 0 0) 0x7f454c0b7c00 con 0x7f454c1430c0 2024-03-25T19:45:45.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.101+0000 7f453affd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:45.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.101+0000 7f453affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:45.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.101+0000 7f453affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:45.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.101+0000 7f453affd640 10 monclient(hunting): my global_id is 4679 2024-03-25T19:45:45.101 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.101+0000 7f453affd640 1 -- 172.21.15.44:0/2642299030 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f452c004eb0 con 0x7f454c1430c0 2024-03-25T19:45:45.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.101+0000 7f453affd640 1 -- 172.21.15.44:0/2642299030 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723602344 0 0) 0x7f454c13b180 con 0x7f454c0a98f0 2024-03-25T19:45:45.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.102+0000 7f453affd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:45.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.102+0000 7f453affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:45.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.102+0000 7f453affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:45.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.102+0000 7f453affd640 10 monclient(hunting): my global_id is 4695 2024-03-25T19:45:45.102 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.102+0000 7f453affd640 1 -- 172.21.15.44:0/2642299030 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f454c0b7c00 con 0x7f454c0a98f0 2024-03-25T19:45:45.105 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.105+0000 7f453affd640 1 -- 172.21.15.44:0/2642299030 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720561806 0 0) 0x7f454c12b950 con 0x7f454c13f710 2024-03-25T19:45:45.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.105+0000 7f453affd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:45.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.105+0000 7f453affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:45.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.105+0000 7f453affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:45.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.105+0000 7f453affd640 10 monclient(hunting): my global_id is 4687 2024-03-25T19:45:45.106 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.105+0000 7f453affd640 1 -- 172.21.15.44:0/2642299030 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f454c13b180 con 0x7f454c13f710 2024-03-25T19:45:45.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.128+0000 7f453affd640 1 -- 172.21.15.44:0/2642299030 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1060312463 0 0) 0x7f454c13b180 con 0x7f454c13f710 2024-03-25T19:45:45.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.128+0000 7f453affd640 1 -- 172.21.15.44:0/2642299030 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f454c12b950 con 0x7f454c13f710 2024-03-25T19:45:45.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.128+0000 7f453affd640 1 -- 172.21.15.44:0/2642299030 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f4540001950 con 0x7f454c13f710 2024-03-25T19:45:45.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.128+0000 7f453affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:45.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.128+0000 7f453affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:45.129 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.128+0000 7f453affd640 10 monclient(hunting): dump: 2024-03-25T19:45:45.129 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:45.129 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:45.129 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:45.129 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:45.129 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:45.129 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:45.129 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:45.130 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:45.130 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:45.130 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:45.130 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.128+0000 7f453affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:45.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.130+0000 7f453affd640 1 -- 172.21.15.44:0/2642299030 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2938442312 0 0) 0x7f454c0b7c00 con 0x7f454c0a98f0 2024-03-25T19:45:45.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.130+0000 7f453affd640 1 -- 172.21.15.44:0/2642299030 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f454c13b180 con 0x7f454c0a98f0 2024-03-25T19:45:45.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.130+0000 7f453affd640 1 -- 172.21.15.44:0/2642299030 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f453c001ae0 con 0x7f454c0a98f0 2024-03-25T19:45:45.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.130+0000 7f453affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:45.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.130+0000 7f453affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:45.131 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.130+0000 7f453affd640 10 monclient(hunting): dump: 2024-03-25T19:45:45.131 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:45.131 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:45.131 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:45.131 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:45.131 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:45.131 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:45.132 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:45.132 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:45.132 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:45.132 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:45.132 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.130+0000 7f453affd640 1 -- 172.21.15.44:0/2642299030 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3609037670 0 0) 0x7f452c004eb0 con 0x7f454c1430c0 2024-03-25T19:45:45.132 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.130+0000 7f453affd640 1 -- 172.21.15.44:0/2642299030 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f454c0b7c00 con 0x7f454c1430c0 2024-03-25T19:45:45.132 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.130+0000 7f453affd640 1 -- 172.21.15.44:0/2642299030 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f4544001b20 con 0x7f454c1430c0 2024-03-25T19:45:45.132 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.130+0000 7f453affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:45.132 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.130+0000 7f453affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:45.132 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.130+0000 7f453affd640 10 monclient(hunting): dump: 2024-03-25T19:45:45.132 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:45.132 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:45.132 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:45.133 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:45.133 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:45.133 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:45.133 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:45.133 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:45.133 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:45.133 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:45.143 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.142+0000 7f453affd640 1 -- 172.21.15.44:0/2642299030 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 424463734 0 0) 0x7f454c13b180 con 0x7f454c0a98f0 2024-03-25T19:45:45.143 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.142+0000 7f453affd640 1 -- 172.21.15.44:0/2642299030 >> v1:172.21.15.44:6790/0 conn(0x7f454c1430c0 legacy=0x7f454c12b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:45.143 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.142+0000 7f453affd640 1 -- 172.21.15.44:0/2642299030 >> v1:172.21.15.111:6789/0 conn(0x7f454c13f710 legacy=0x7f454c12ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:45.143 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.142+0000 7f453affd640 10 monclient: _finish_hunting 0 2024-03-25T19:45:45.143 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.142+0000 7f453affd640 1 monclient: found mon.a 2024-03-25T19:45:45.143 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.142+0000 7f453affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:45.143 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.142+0000 7f453affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:45:45.143 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.142+0000 7f453affd640 1 -- 172.21.15.44:0/2642299030 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f454c13b3f0 con 0x7f454c0a98f0 2024-03-25T19:45:45.143 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.142+0000 7f453affd640 10 monclient: _finish_auth 0 2024-03-25T19:45:45.143 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.142+0000 7f453affd640 10 monclient: _check_auth_tickets 2024-03-25T19:45:45.143 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.142+0000 7f453affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:45.143 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.142+0000 7f4550d13640 5 monclient: authenticate success, global_id 4695 2024-03-25T19:45:45.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.142+0000 7f4550d13640 10 monclient: _renew_subs 2024-03-25T19:45:45.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.142+0000 7f4550d13640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:45:45.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.142+0000 7f4550d13640 1 -- 172.21.15.44:0/2642299030 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f454c13b6d0 con 0x7f454c0a98f0 2024-03-25T19:45:45.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.142+0000 7f4550d13640 10 client.4695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:45.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.142+0000 7f4550d13640 10 monclient: _renew_subs 2024-03-25T19:45:45.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.142+0000 7f4550d13640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:45:45.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.142+0000 7f4550d13640 1 -- 172.21.15.44:0/2642299030 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f454c13a200 con 0x7f454c0a98f0 2024-03-25T19:45:45.144 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.142+0000 7f4550d13640 1 librados: init done 2024-03-25T19:45:45.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.143+0000 7f4550d13640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:45.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.143+0000 7f4550d13640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:45.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.143+0000 7f4550d13640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:45:45.145 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.143+0000 7f4550d13640 1 -- 172.21.15.44:0/2642299030 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f454c074cc0 con 0x7f454c0a98f0 2024-03-25T19:45:45.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.461+0000 7f453affd640 1 -- 172.21.15.44:0/2642299030 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f453c0038a0 con 0x7f454c0a98f0 2024-03-25T19:45:45.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.461+0000 7f453affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:45.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.461+0000 7f453affd640 1 -- 172.21.15.44:0/2642299030 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f453c003bb0 con 0x7f454c0a98f0 2024-03-25T19:45:45.461 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.461+0000 7f453affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:45.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.461+0000 7f453affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:45.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.461+0000 7f453affd640 10 monclient: dump: 2024-03-25T19:45:45.462 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:45.462 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:45.462 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:45.462 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:45.462 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:45.462 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:45.462 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:45.462 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:45.462 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:45.462 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:45.462 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.461+0000 7f4548d99640 4 set_mon_vals no callback set 2024-03-25T19:45:45.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.462+0000 7f453affd640 1 -- 172.21.15.44:0/2642299030 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f453c019b80 con 0x7f454c0a98f0 2024-03-25T19:45:45.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.463+0000 7f4549d9b640 10 monclient: get_auth_request con 0x7f452c06e160 auth_method 0 2024-03-25T19:45:45.463 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.463+0000 7f453affd640 10 client.4695.objecter ms_handle_connect 0x7f452c06e160 2024-03-25T19:45:45.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.468+0000 7f453affd640 1 -- 172.21.15.44:0/2642299030 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1275227418 0 0) 0x7f453c053660 con 0x7f454c0a98f0 2024-03-25T19:45:45.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.468+0000 7f453affd640 10 client.4695.objecter ms_dispatch 0x7f454c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:45:45.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.468+0000 7f453affd640 3 client.4695.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:45:45.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.468+0000 7f453affd640 3 client.4695.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:45:45.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.468+0000 7f453affd640 20 client.4695.objecter dump_active .. 0 homeless 2024-03-25T19:45:45.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.468+0000 7f453affd640 1 -- 172.21.15.44:0/2642299030 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f453c04f170 con 0x7f454c0a98f0 2024-03-25T19:45:45.468 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.468+0000 7f453affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:45.469 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.468+0000 7f453affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:45.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.699+0000 7f4550d13640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:45.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.699+0000 7f4550d13640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:45.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.699+0000 7f4550d13640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:45:45.700 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.699+0000 7f4550d13640 1 -- 172.21.15.44:0/2642299030 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f454c13a500 con 0x7f454c0a98f0 2024-03-25T19:45:45.709 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:10:36.610 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:45.718 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:10:36.620 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:45.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.743+0000 7f453affd640 1 -- 172.21.15.44:0/2642299030 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2782837909) 0x7f453c0510a0 con 0x7f454c0a98f0 2024-03-25T19:45:45.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.743+0000 7f453affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:45.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.743+0000 7f453affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:45.743 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:45:45.743 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":11,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:45:30.548832+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:45:30.548832+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:45.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.746+0000 7f4550d13640 10 librados: watch_flush enter 2024-03-25T19:45:45.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.746+0000 7f4550d13640 10 librados: watch_flush exit 2024-03-25T19:45:45.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.746+0000 7f4550d13640 20 client.4695.objecter shutdown clearing up homeless session... 2024-03-25T19:45:45.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.746+0000 7f4550d13640 10 client.4695.objecter successfully canceled tick 2024-03-25T19:45:45.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.746+0000 7f4550d13640 1 -- 172.21.15.44:0/2642299030 >> v1:172.21.15.111:6800/3883215793 conn(0x7f452c06e160 legacy=0x7f452c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:45.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.746+0000 7f4550d13640 10 monclient: shutdown 2024-03-25T19:45:45.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.746+0000 7f4550d13640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:45.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.746+0000 7f4550d13640 1 -- 172.21.15.44:0/2642299030 >> v1:172.21.15.44:6789/0 conn(0x7f454c0a98f0 legacy=0x7f454c0b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:45.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.746+0000 7f4550d13640 1 -- 172.21.15.44:0/2642299030 shutdown_connections 2024-03-25T19:45:45.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.746+0000 7f4550d13640 1 -- 172.21.15.44:0/2642299030 >> 172.21.15.44:0/2642299030 conn(0x7f454c0b7ea0 msgr2=0x7f454c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:45.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.746+0000 7f4550d13640 1 -- 172.21.15.44:0/2642299030 shutdown_connections 2024-03-25T19:45:45.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.746+0000 7f4550d13640 1 -- 172.21.15.44:0/2642299030 wait complete. 2024-03-25T19:45:45.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:45.746+0000 7f4550d13640 1 librados: shutdown 2024-03-25T19:45:45.756 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:45.774 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:10:36.676 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:45.777 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:10:36.679 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:45.785 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:10:36.686 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:45.798 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:10:36.678 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:45.812 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:10:36.691 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:45.821 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:10:36.723 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:45.853 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:10:36.733 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:52.563 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:52.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.668+0000 7f1b7ab54640 10 monclient: get_monmap_and_config 2024-03-25T19:45:52.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.668+0000 7f1b7ab54640 10 monclient: build_initial_monmap 2024-03-25T19:45:52.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.668+0000 7f1b7ab54640 10 monclient: monmap: 2024-03-25T19:45:52.669 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:45:52.669 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:52.669 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:45:52.669480+0000 2024-03-25T19:45:52.669 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:45:52.669480+0000 2024-03-25T19:45:52.669 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:52.669 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:52.670 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:45:52.670 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:45:52.670 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:45:52.670 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:52.671 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.669+0000 7f1b7ab54640 1 Processor -- start 2024-03-25T19:45:52.671 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.669+0000 7f1b7ab54640 1 -- start start 2024-03-25T19:45:52.671 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.669+0000 7f1b7ab54640 10 monclient: init 2024-03-25T19:45:52.671 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.669+0000 7f1b7ab54640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:52.671 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.669+0000 7f1b7ab54640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:52.671 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.669+0000 7f1b7ab54640 10 monclient(hunting): picked mon.noname-a con 0x7f1b740a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:45:52.671 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.669+0000 7f1b7ab54640 10 monclient(hunting): picked mon.noname-c con 0x7f1b7413f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:45:52.671 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.669+0000 7f1b7ab54640 10 monclient(hunting): picked mon.noname-b con 0x7f1b741430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:45:52.671 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.669+0000 7f1b7ab54640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b74142520 con 0x7f1b740a98f0 2024-03-25T19:45:52.671 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.670+0000 7f1b7ab54640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b740b28a0 con 0x7f1b7413f710 2024-03-25T19:45:52.672 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.670+0000 7f1b7ab54640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b740b2a30 con 0x7f1b741430c0 2024-03-25T19:45:52.672 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.670+0000 7f1b7ab54640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:52.672 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.670+0000 7f1b7ab54640 10 monclient(hunting): authenticate will time out at 6016.840856s 2024-03-25T19:45:52.672 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.671+0000 7f1b73fff640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f1b7413f710 0x7f1b741429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:60334/0 (socket says 172.21.15.44:60334) 2024-03-25T19:45:52.672 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.671+0000 7f1b73fff640 1 -- 172.21.15.44:0/2625947621 learned_addr learned my addr 172.21.15.44:0/2625947621 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:45:52.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.680+0000 7f1b737fe640 1 -- 172.21.15.44:0/2625947621 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3686849514 0 0) 0x7f1b740b28a0 con 0x7f1b7413f710 2024-03-25T19:45:52.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.680+0000 7f1b737fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:52.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.680+0000 7f1b737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:52.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.680+0000 7f1b737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:52.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.680+0000 7f1b737fe640 10 monclient(hunting): my global_id is 4690 2024-03-25T19:45:52.682 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.680+0000 7f1b737fe640 1 -- 172.21.15.44:0/2625947621 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b64003520 con 0x7f1b7413f710 2024-03-25T19:45:52.689 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.689+0000 7f1b737fe640 1 -- 172.21.15.44:0/2625947621 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 960981536 0 0) 0x7f1b64003520 con 0x7f1b7413f710 2024-03-25T19:45:52.689 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.689+0000 7f1b737fe640 1 -- 172.21.15.44:0/2625947621 >> v1:172.21.15.44:6790/0 conn(0x7f1b741430c0 legacy=0x7f1b7414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:52.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.689+0000 7f1b737fe640 1 -- 172.21.15.44:0/2625947621 >> v1:172.21.15.44:6789/0 conn(0x7f1b740a98f0 legacy=0x7f1b740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:45:52.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.689+0000 7f1b737fe640 10 monclient: _finish_hunting 0 2024-03-25T19:45:52.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.689+0000 7f1b737fe640 1 monclient: found mon.noname-c 2024-03-25T19:45:52.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.689+0000 7f1b737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:52.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.689+0000 7f1b737fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:45:52.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.689+0000 7f1b737fe640 1 -- 172.21.15.44:0/2625947621 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b740b44c0 con 0x7f1b7413f710 2024-03-25T19:45:52.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.689+0000 7f1b737fe640 10 monclient: _finish_auth 0 2024-03-25T19:45:52.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.689+0000 7f1b737fe640 10 monclient: _check_auth_tickets 2024-03-25T19:45:52.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.689+0000 7f1b737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:52.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.689+0000 7f1b737fe640 1 -- 172.21.15.44:0/2625947621 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f1b60001690 con 0x7f1b7413f710 2024-03-25T19:45:52.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.689+0000 7f1b737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:52.690 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.689+0000 7f1b737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:52.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.689+0000 7f1b737fe640 10 monclient: dump: 2024-03-25T19:45:52.691 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:52.691 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:52.691 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:52.691 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:52.691 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:52.691 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:52.691 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:52.691 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:52.691 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:52.691 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:52.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.689+0000 7f1b7ab54640 5 monclient: authenticate success, global_id 4690 2024-03-25T19:45:52.691 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.689+0000 7f1b7ab54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:52.697 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.697+0000 7f1b737fe640 1 -- 172.21.15.44:0/2625947621 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b60001860 con 0x7f1b7413f710 2024-03-25T19:45:52.697 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.697+0000 7f1b737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:52.697 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.697+0000 7f1b7ab54640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:52.697 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.697+0000 7f1b7ab54640 4 set_mon_vals no callback set 2024-03-25T19:45:52.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.697+0000 7f1b737fe640 1 -- 172.21.15.44:0/2625947621 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f1b60001cb0 con 0x7f1b7413f710 2024-03-25T19:45:52.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.697+0000 7f1b737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:52.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.697+0000 7f1b737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:52.698 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.697+0000 7f1b737fe640 10 monclient: dump: 2024-03-25T19:45:52.698 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:52.698 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:52.698 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:52.699 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:52.699 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:52.699 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:52.699 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:52.699 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:52.699 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:52.699 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:52.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.697+0000 7f1b7ab54640 10 monclient: shutdown 2024-03-25T19:45:52.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.697+0000 7f1b7ab54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:52.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.697+0000 7f1b7ab54640 1 -- 172.21.15.44:0/2625947621 >> v1:172.21.15.111:6789/0 conn(0x7f1b7413f710 legacy=0x7f1b741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:52.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.697+0000 7f1b7ab54640 1 -- 172.21.15.44:0/2625947621 shutdown_connections 2024-03-25T19:45:52.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.697+0000 7f1b7ab54640 1 -- 172.21.15.44:0/2625947621 >> 172.21.15.44:0/2625947621 conn(0x7f1b740b7ea0 msgr2=0x7f1b740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:52.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.698+0000 7f1b7ab54640 1 -- 172.21.15.44:0/2625947621 shutdown_connections 2024-03-25T19:45:52.699 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.698+0000 7f1b7ab54640 1 -- 172.21.15.44:0/2625947621 wait complete. 2024-03-25T19:45:52.701 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.698+0000 7f1b7ab54640 10 monclient: build_initial_monmap 2024-03-25T19:45:52.701 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.698+0000 7f1b7ab54640 10 monclient: monmap: 2024-03-25T19:45:52.701 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:45:52.701 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:52.701 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:45:52.701 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:45:52.702 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:52.702 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:45:52.702 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:45:52.702 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:45:52.702 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:45:52.702 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:52.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.699+0000 7f1b7ab54640 1 librados: starting msgr at 2024-03-25T19:45:52.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.699+0000 7f1b7ab54640 1 librados: starting objecter 2024-03-25T19:45:52.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.699+0000 7f1b7ab54640 1 Processor -- start 2024-03-25T19:45:52.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.699+0000 7f1b7ab54640 1 -- start start 2024-03-25T19:45:52.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.699+0000 7f1b7ab54640 1 librados: setting wanted keys 2024-03-25T19:45:52.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.699+0000 7f1b7ab54640 1 librados: calling monclient init 2024-03-25T19:45:52.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.699+0000 7f1b7ab54640 10 monclient: init 2024-03-25T19:45:52.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.700+0000 7f1b7ab54640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:52.702 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.700+0000 7f1b7ab54640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:45:52.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.700+0000 7f1b7ab54640 10 monclient(hunting): picked mon.noname-b con 0x7f1b740a98f0 addr v1:172.21.15.111:6789/0 2024-03-25T19:45:52.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.700+0000 7f1b7ab54640 10 monclient(hunting): picked mon.noname-c con 0x7f1b7413f710 addr v1:172.21.15.44:6790/0 2024-03-25T19:45:52.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.700+0000 7f1b7ab54640 10 monclient(hunting): picked mon.noname-a con 0x7f1b741430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:45:52.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.700+0000 7f1b7ab54640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b74139030 con 0x7f1b741430c0 2024-03-25T19:45:52.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.700+0000 7f1b7ab54640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b740abde0 con 0x7f1b740a98f0 2024-03-25T19:45:52.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.700+0000 7f1b7ab54640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b740ac0c0 con 0x7f1b7413f710 2024-03-25T19:45:52.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.700+0000 7f1b7ab54640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:52.703 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.700+0000 7f1b7ab54640 10 monclient(hunting): authenticate will time out at 6016.871823s 2024-03-25T19:45:52.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.708+0000 7f1b788f0640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f1b740a98f0 0x7f1b74138920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:60366/0 (socket says 172.21.15.44:60366) 2024-03-25T19:45:52.709 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.708+0000 7f1b788f0640 1 -- 172.21.15.44:0/1369491996 learned_addr learned my addr 172.21.15.44:0/1369491996 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:45:52.711 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.711+0000 7f1b717fa640 10 client.?.objecter ms_handle_connect 0x7f1b740a98f0 2024-03-25T19:45:52.711 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.711+0000 7f1b717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:52.719 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.718+0000 7f1b717fa640 1 -- 172.21.15.44:0/1369491996 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400871968 0 0) 0x7f1b740abde0 con 0x7f1b740a98f0 2024-03-25T19:45:52.719 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.718+0000 7f1b717fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:52.719 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.718+0000 7f1b717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:52.719 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.718+0000 7f1b717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:52.719 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.718+0000 7f1b717fa640 10 monclient(hunting): my global_id is 4693 2024-03-25T19:45:52.719 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.718+0000 7f1b717fa640 1 -- 172.21.15.44:0/1369491996 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b58004eb0 con 0x7f1b740a98f0 2024-03-25T19:45:52.722 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.722+0000 7f1b717fa640 10 client.?.objecter ms_handle_connect 0x7f1b7413f710 2024-03-25T19:45:52.722 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.722+0000 7f1b717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:52.734 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.734+0000 7f1b717fa640 1 -- 172.21.15.44:0/1369491996 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4283965934 0 0) 0x7f1b740ac0c0 con 0x7f1b7413f710 2024-03-25T19:45:52.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.734+0000 7f1b717fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:52.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.734+0000 7f1b717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:52.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.734+0000 7f1b717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:52.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.734+0000 7f1b717fa640 10 monclient(hunting): my global_id is 4685 2024-03-25T19:45:52.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.734+0000 7f1b717fa640 1 -- 172.21.15.44:0/1369491996 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b740abde0 con 0x7f1b7413f710 2024-03-25T19:45:52.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.734+0000 7f1b717fa640 10 client.?.objecter ms_handle_connect 0x7f1b741430c0 2024-03-25T19:45:52.735 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.734+0000 7f1b717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:52.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.743+0000 7f1b717fa640 1 -- 172.21.15.44:0/1369491996 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3371465792 0 0) 0x7f1b58004eb0 con 0x7f1b740a98f0 2024-03-25T19:45:52.743 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.743+0000 7f1b717fa640 1 -- 172.21.15.44:0/1369491996 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b740ac0c0 con 0x7f1b740a98f0 2024-03-25T19:45:52.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.743+0000 7f1b717fa640 1 -- 172.21.15.44:0/1369491996 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f1b68001960 con 0x7f1b740a98f0 2024-03-25T19:45:52.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.743+0000 7f1b717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:52.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.743+0000 7f1b717fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:52.744 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.743+0000 7f1b717fa640 10 monclient(hunting): dump: 2024-03-25T19:45:52.744 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:52.744 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:52.744 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:52.744 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:52.744 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:52.744 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:52.744 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:52.744 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:52.744 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:52.744 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:52.745 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.743+0000 7f1b717fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:52.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.747+0000 7f1b717fa640 1 -- 172.21.15.44:0/1369491996 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910342066 0 0) 0x7f1b74139030 con 0x7f1b741430c0 2024-03-25T19:45:52.747 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.747+0000 7f1b717fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:52.748 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.747+0000 7f1b717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:52.748 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.747+0000 7f1b717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:52.748 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.747+0000 7f1b717fa640 10 monclient(hunting): my global_id is 4698 2024-03-25T19:45:52.748 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.747+0000 7f1b717fa640 1 -- 172.21.15.44:0/1369491996 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b58004eb0 con 0x7f1b741430c0 2024-03-25T19:45:52.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.748+0000 7f1b717fa640 1 -- 172.21.15.44:0/1369491996 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3985996630 0 0) 0x7f1b740ac0c0 con 0x7f1b740a98f0 2024-03-25T19:45:52.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.748+0000 7f1b717fa640 1 -- 172.21.15.44:0/1369491996 >> v1:172.21.15.44:6790/0 conn(0x7f1b7413f710 legacy=0x7f1b740ac4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:52.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.748+0000 7f1b717fa640 1 -- 172.21.15.44:0/1369491996 >> v1:172.21.15.44:6789/0 conn(0x7f1b741430c0 legacy=0x7f1b741e25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:52.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.748+0000 7f1b717fa640 10 monclient: _finish_hunting 0 2024-03-25T19:45:52.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.748+0000 7f1b717fa640 1 monclient: found mon.b 2024-03-25T19:45:52.749 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.748+0000 7f1b717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:52.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.748+0000 7f1b717fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:45:52.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.748+0000 7f1b717fa640 1 -- 172.21.15.44:0/1369491996 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b740acbf0 con 0x7f1b740a98f0 2024-03-25T19:45:52.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.748+0000 7f1b717fa640 10 monclient: _finish_auth 0 2024-03-25T19:45:52.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.748+0000 7f1b717fa640 10 monclient: _check_auth_tickets 2024-03-25T19:45:52.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.748+0000 7f1b717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:52.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.748+0000 7f1b7ab54640 5 monclient: authenticate success, global_id 4693 2024-03-25T19:45:52.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.748+0000 7f1b7ab54640 10 monclient: _renew_subs 2024-03-25T19:45:52.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.748+0000 7f1b7ab54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:45:52.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.748+0000 7f1b7ab54640 1 -- 172.21.15.44:0/1369491996 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b74139260 con 0x7f1b740a98f0 2024-03-25T19:45:52.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.748+0000 7f1b7ab54640 10 client.4693.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:52.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.748+0000 7f1b7ab54640 10 monclient: _renew_subs 2024-03-25T19:45:52.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.748+0000 7f1b7ab54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:45:52.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.748+0000 7f1b7ab54640 1 -- 172.21.15.44:0/1369491996 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b74137f80 con 0x7f1b740a98f0 2024-03-25T19:45:52.750 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.748+0000 7f1b7ab54640 1 librados: init done 2024-03-25T19:45:52.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.749+0000 7f1b7ab54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:52.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.749+0000 7f1b7ab54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:52.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.749+0000 7f1b7ab54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:45:52.751 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:52.749+0000 7f1b7ab54640 1 -- 172.21.15.44:0/1369491996 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b7414f2d0 con 0x7f1b740a98f0 2024-03-25T19:45:53.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.031+0000 7f1b717fa640 1 -- 172.21.15.44:0/1369491996 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b680035f0 con 0x7f1b740a98f0 2024-03-25T19:45:53.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.031+0000 7f1b717fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:53.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.031+0000 7f1b717fa640 1 -- 172.21.15.44:0/1369491996 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f1b68003940 con 0x7f1b740a98f0 2024-03-25T19:45:53.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.031+0000 7f1b717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:53.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.031+0000 7f1b717fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:53.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.031+0000 7f1b717fa640 10 monclient: dump: 2024-03-25T19:45:53.032 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:45:53.032 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:45:53.032 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:53.032 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:45:53.032 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:53.032 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:45:53.032 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:45:53.032 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:45:53.033 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:45:53.033 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:45:53.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.031+0000 7f1b72ffd640 4 set_mon_vals no callback set 2024-03-25T19:45:53.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.032+0000 7f1b717fa640 1 -- 172.21.15.44:0/1369491996 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f1b68003b20 con 0x7f1b740a98f0 2024-03-25T19:45:53.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.032+0000 7f1b73fff640 10 monclient: get_auth_request con 0x7f1b5806ddb0 auth_method 0 2024-03-25T19:45:53.034 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.033+0000 7f1b717fa640 10 client.4693.objecter ms_handle_connect 0x7f1b5806ddb0 2024-03-25T19:45:53.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.037+0000 7f1b717fa640 1 -- 172.21.15.44:0/1369491996 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1275227418 0 0) 0x7f1b68054830 con 0x7f1b740a98f0 2024-03-25T19:45:53.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.037+0000 7f1b717fa640 10 client.4693.objecter ms_dispatch 0x7f1b74000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:45:53.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.037+0000 7f1b717fa640 3 client.4693.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:45:53.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.037+0000 7f1b717fa640 3 client.4693.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:45:53.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.037+0000 7f1b717fa640 20 client.4693.objecter dump_active .. 0 homeless 2024-03-25T19:45:53.038 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.038+0000 7f1b717fa640 1 -- 172.21.15.44:0/1369491996 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b68050340 con 0x7f1b740a98f0 2024-03-25T19:45:53.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.038+0000 7f1b717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:53.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.038+0000 7f1b717fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:53.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.291+0000 7f1b7ab54640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:53.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.291+0000 7f1b7ab54640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:53.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.291+0000 7f1b7ab54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:45:53.291 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.291+0000 7f1b7ab54640 1 -- 172.21.15.44:0/1369491996 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b740627b0 con 0x7f1b740a98f0 2024-03-25T19:45:53.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.309+0000 7f1b717fa640 1 -- 172.21.15.44:0/1369491996 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2782837909) 0x7f1b68052270 con 0x7f1b740a98f0 2024-03-25T19:45:53.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.309+0000 7f1b717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:53.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.309+0000 7f1b717fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:53.309 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:45:53.309 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":11,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:45:30.548832+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:45:30.548832+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:53.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.311+0000 7f1b7ab54640 10 librados: watch_flush enter 2024-03-25T19:45:53.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.311+0000 7f1b7ab54640 10 librados: watch_flush exit 2024-03-25T19:45:53.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.311+0000 7f1b7ab54640 20 client.4693.objecter shutdown clearing up homeless session... 2024-03-25T19:45:53.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.311+0000 7f1b7ab54640 10 client.4693.objecter successfully canceled tick 2024-03-25T19:45:53.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.311+0000 7f1b7ab54640 1 -- 172.21.15.44:0/1369491996 >> v1:172.21.15.111:6800/3883215793 conn(0x7f1b5806ddb0 legacy=0x7f1b58070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:53.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.311+0000 7f1b7ab54640 10 monclient: shutdown 2024-03-25T19:45:53.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.311+0000 7f1b7ab54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:53.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.311+0000 7f1b7ab54640 1 -- 172.21.15.44:0/1369491996 >> v1:172.21.15.111:6789/0 conn(0x7f1b740a98f0 legacy=0x7f1b74138920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:53.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.311+0000 7f1b7ab54640 1 -- 172.21.15.44:0/1369491996 shutdown_connections 2024-03-25T19:45:53.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.311+0000 7f1b7ab54640 1 -- 172.21.15.44:0/1369491996 >> 172.21.15.44:0/1369491996 conn(0x7f1b740b7ea0 msgr2=0x7f1b74074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:53.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.311+0000 7f1b7ab54640 1 -- 172.21.15.44:0/1369491996 shutdown_connections 2024-03-25T19:45:53.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.311+0000 7f1b7ab54640 1 -- 172.21.15.44:0/1369491996 wait complete. 2024-03-25T19:45:53.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:45:53.311+0000 7f1b7ab54640 1 librados: shutdown 2024-03-25T19:45:53.324 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:00.131 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:00.249 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.249+0000 7f0caa938640 10 monclient: get_monmap_and_config 2024-03-25T19:46:00.249 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.249+0000 7f0caa938640 10 monclient: build_initial_monmap 2024-03-25T19:46:00.249 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.249+0000 7f0caa938640 10 monclient: monmap: 2024-03-25T19:46:00.249 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:46:00.249 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:00.250 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:46:00.250430+0000 2024-03-25T19:46:00.250 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:46:00.250430+0000 2024-03-25T19:46:00.250 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:00.250 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:00.250 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:46:00.250 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:46:00.250 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:46:00.250 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:00.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.250+0000 7f0caa938640 1 Processor -- start 2024-03-25T19:46:00.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.250+0000 7f0caa938640 1 -- start start 2024-03-25T19:46:00.251 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.250+0000 7f0caa938640 10 monclient: init 2024-03-25T19:46:00.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.251+0000 7f0caa938640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:00.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.251+0000 7f0caa938640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:46:00.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.251+0000 7f0caa938640 10 monclient(hunting): picked mon.noname-c con 0x7f0ca40b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T19:46:00.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.251+0000 7f0caa938640 10 monclient(hunting): picked mon.noname-b con 0x7f0ca40b1730 addr v1:172.21.15.44:6790/0 2024-03-25T19:46:00.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.251+0000 7f0caa938640 10 monclient(hunting): picked mon.noname-a con 0x7f0ca414be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:46:00.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.251+0000 7f0caa938640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca40af940 con 0x7f0ca414be90 2024-03-25T19:46:00.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.251+0000 7f0caa938640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca40b0a70 con 0x7f0ca40b4d60 2024-03-25T19:46:00.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.251+0000 7f0caa938640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca40b0d50 con 0x7f0ca40b1730 2024-03-25T19:46:00.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.251+0000 7f0caa938640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:00.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.251+0000 7f0caa938640 10 monclient(hunting): authenticate will time out at 6024.422023s 2024-03-25T19:46:00.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.265+0000 7f0ca8ed5640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f0ca414be90 0x7f0ca414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41938/0 (socket says 172.21.15.44:41938) 2024-03-25T19:46:00.265 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.265+0000 7f0ca8ed5640 1 -- 172.21.15.44:0/1577547376 learned_addr learned my addr 172.21.15.44:0/1577547376 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:46:00.274 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.274+0000 7f0ca2ffd640 1 -- 172.21.15.44:0/1577547376 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156113907 0 0) 0x7f0ca40b0d50 con 0x7f0ca40b1730 2024-03-25T19:46:00.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.274+0000 7f0ca2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:00.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.274+0000 7f0ca2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:00.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.274+0000 7f0ca2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:00.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.274+0000 7f0ca2ffd640 10 monclient(hunting): my global_id is 4688 2024-03-25T19:46:00.275 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.274+0000 7f0ca2ffd640 1 -- 172.21.15.44:0/1577547376 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c8c003520 con 0x7f0ca40b1730 2024-03-25T19:46:00.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.279+0000 7f0ca2ffd640 1 -- 172.21.15.44:0/1577547376 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606600781 0 0) 0x7f0ca40af940 con 0x7f0ca414be90 2024-03-25T19:46:00.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.279+0000 7f0ca2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:00.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.279+0000 7f0ca2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:00.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.279+0000 7f0ca2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:00.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.279+0000 7f0ca2ffd640 10 monclient(hunting): my global_id is 4701 2024-03-25T19:46:00.279 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.279+0000 7f0ca2ffd640 1 -- 172.21.15.44:0/1577547376 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ca40b0d50 con 0x7f0ca414be90 2024-03-25T19:46:00.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.281+0000 7f0ca2ffd640 1 -- 172.21.15.44:0/1577547376 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2907796309 0 0) 0x7f0ca40b0a70 con 0x7f0ca40b4d60 2024-03-25T19:46:00.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.281+0000 7f0ca2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:00.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.281+0000 7f0ca2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:00.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.281+0000 7f0ca2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:00.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.281+0000 7f0ca2ffd640 10 monclient(hunting): my global_id is 4696 2024-03-25T19:46:00.281 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.281+0000 7f0ca2ffd640 1 -- 172.21.15.44:0/1577547376 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ca40af940 con 0x7f0ca40b4d60 2024-03-25T19:46:00.294 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.293+0000 7f0ca2ffd640 1 -- 172.21.15.44:0/1577547376 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 818092277 0 0) 0x7f0ca40b0d50 con 0x7f0ca414be90 2024-03-25T19:46:00.294 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.293+0000 7f0ca2ffd640 1 -- 172.21.15.44:0/1577547376 >> v1:172.21.15.44:6790/0 conn(0x7f0ca40b1730 legacy=0x7f0ca40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:00.294 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.293+0000 7f0ca2ffd640 1 -- 172.21.15.44:0/1577547376 >> v1:172.21.15.111:6789/0 conn(0x7f0ca40b4d60 legacy=0x7f0ca413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:00.294 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.293+0000 7f0ca2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:00.294 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.293+0000 7f0ca2ffd640 1 monclient: found mon.noname-a 2024-03-25T19:46:00.294 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.293+0000 7f0ca2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:00.294 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.293+0000 7f0ca2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:46:00.294 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.293+0000 7f0ca2ffd640 1 -- 172.21.15.44:0/1577547376 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca40b1030 con 0x7f0ca414be90 2024-03-25T19:46:00.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.293+0000 7f0ca2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:46:00.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.293+0000 7f0ca2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:00.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.293+0000 7f0ca2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:00.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.293+0000 7f0ca2ffd640 1 -- 172.21.15.44:0/1577547376 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f0c9c0019f0 con 0x7f0ca414be90 2024-03-25T19:46:00.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.293+0000 7f0caa938640 5 monclient: authenticate success, global_id 4701 2024-03-25T19:46:00.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.293+0000 7f0caa938640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:00.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.293+0000 7f0ca2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:00.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.293+0000 7f0ca2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:00.295 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.293+0000 7f0ca2ffd640 10 monclient: dump: 2024-03-25T19:46:00.295 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:00.295 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:00.295 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:00.295 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:00.295 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:00.296 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:00.296 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:00.296 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:00.296 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:00.296 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:00.296 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.293+0000 7f0caa938640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:00.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.316+0000 7f0ca2ffd640 1 -- 172.21.15.44:0/1577547376 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c9c001b50 con 0x7f0ca414be90 2024-03-25T19:46:00.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.316+0000 7f0ca2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:00.316 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.316+0000 7f0ca2ffd640 1 -- 172.21.15.44:0/1577547376 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f0c9c001f60 con 0x7f0ca414be90 2024-03-25T19:46:00.317 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.316+0000 7f0caa938640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:00.317 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.316+0000 7f0caa938640 4 set_mon_vals no callback set 2024-03-25T19:46:00.317 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.316+0000 7f0ca2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:00.317 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.316+0000 7f0ca2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:00.317 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.316+0000 7f0ca2ffd640 10 monclient: dump: 2024-03-25T19:46:00.317 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:00.317 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:00.317 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:00.317 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:00.317 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:00.317 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:00.317 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:00.317 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:00.317 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:00.317 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:00.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.316+0000 7f0caa938640 10 monclient: shutdown 2024-03-25T19:46:00.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.316+0000 7f0caa938640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:00.320 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.316+0000 7f0caa938640 1 -- 172.21.15.44:0/1577547376 >> v1:172.21.15.44:6789/0 conn(0x7f0ca414be90 legacy=0x7f0ca414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:00.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.317+0000 7f0caa938640 1 -- 172.21.15.44:0/1577547376 shutdown_connections 2024-03-25T19:46:00.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.317+0000 7f0caa938640 1 -- 172.21.15.44:0/1577547376 >> 172.21.15.44:0/1577547376 conn(0x7f0ca40617a0 msgr2=0x7f0ca4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:00.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.317+0000 7f0caa938640 1 -- 172.21.15.44:0/1577547376 shutdown_connections 2024-03-25T19:46:00.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.317+0000 7f0caa938640 1 -- 172.21.15.44:0/1577547376 wait complete. 2024-03-25T19:46:00.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.317+0000 7f0caa938640 10 monclient: build_initial_monmap 2024-03-25T19:46:00.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.317+0000 7f0caa938640 10 monclient: monmap: 2024-03-25T19:46:00.321 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:46:00.321 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:00.321 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:46:00.321 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:46:00.321 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:00.321 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:46:00.321 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:46:00.321 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:46:00.322 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:46:00.322 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:00.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.318+0000 7f0caa938640 1 librados: starting msgr at 2024-03-25T19:46:00.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.318+0000 7f0caa938640 1 librados: starting objecter 2024-03-25T19:46:00.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.318+0000 7f0caa938640 1 Processor -- start 2024-03-25T19:46:00.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.318+0000 7f0caa938640 1 -- start start 2024-03-25T19:46:00.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.318+0000 7f0caa938640 1 librados: setting wanted keys 2024-03-25T19:46:00.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.318+0000 7f0caa938640 1 librados: calling monclient init 2024-03-25T19:46:00.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.318+0000 7f0caa938640 10 monclient: init 2024-03-25T19:46:00.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.319+0000 7f0caa938640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:00.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.319+0000 7f0caa938640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:00.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.319+0000 7f0caa938640 10 monclient(hunting): picked mon.noname-c con 0x7f0ca40b1730 addr v1:172.21.15.44:6790/0 2024-03-25T19:46:00.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.319+0000 7f0caa938640 10 monclient(hunting): picked mon.noname-b con 0x7f0ca40b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T19:46:00.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.319+0000 7f0caa938640 10 monclient(hunting): picked mon.noname-a con 0x7f0ca414be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:46:00.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.319+0000 7f0caa938640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca4142420 con 0x7f0ca414be90 2024-03-25T19:46:00.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.319+0000 7f0caa938640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca4142970 con 0x7f0ca40b4d60 2024-03-25T19:46:00.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.319+0000 7f0caa938640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca41435a0 con 0x7f0ca40b1730 2024-03-25T19:46:00.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.319+0000 7f0caa938640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:00.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.319+0000 7f0caa938640 10 monclient(hunting): authenticate will time out at 6024.490346s 2024-03-25T19:46:00.332 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.331+0000 7f0ca8ed5640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f0ca414be90 0x7f0ca41e1e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:41994/0 (socket says 172.21.15.44:41994) 2024-03-25T19:46:00.332 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.332+0000 7f0ca8ed5640 1 -- 172.21.15.44:0/4265609909 learned_addr learned my addr 172.21.15.44:0/4265609909 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:46:00.333 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.333+0000 7f0ca0ff9640 10 client.?.objecter ms_handle_connect 0x7f0ca414be90 2024-03-25T19:46:00.333 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.333+0000 7f0ca0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:00.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.340+0000 7f0ca0ff9640 10 client.?.objecter ms_handle_connect 0x7f0ca40b4d60 2024-03-25T19:46:00.341 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.340+0000 7f0ca0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:00.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.342+0000 7f0ca0ff9640 10 client.?.objecter ms_handle_connect 0x7f0ca40b1730 2024-03-25T19:46:00.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.342+0000 7f0ca0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:00.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.350+0000 7f0ca0ff9640 1 -- 172.21.15.44:0/4265609909 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2116682568 0 0) 0x7f0ca4142420 con 0x7f0ca414be90 2024-03-25T19:46:00.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.350+0000 7f0ca0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:00.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.350+0000 7f0ca0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:00.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.350+0000 7f0ca0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:00.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.350+0000 7f0ca0ff9640 10 monclient(hunting): my global_id is 4704 2024-03-25T19:46:00.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.350+0000 7f0ca0ff9640 1 -- 172.21.15.44:0/4265609909 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c90004eb0 con 0x7f0ca414be90 2024-03-25T19:46:00.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.355+0000 7f0ca0ff9640 1 -- 172.21.15.44:0/4265609909 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333546175 0 0) 0x7f0ca4142970 con 0x7f0ca40b4d60 2024-03-25T19:46:00.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.355+0000 7f0ca0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:00.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.355+0000 7f0ca0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:00.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.355+0000 7f0ca0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:00.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.355+0000 7f0ca0ff9640 10 monclient(hunting): my global_id is 4699 2024-03-25T19:46:00.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.355+0000 7f0ca0ff9640 1 -- 172.21.15.44:0/4265609909 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ca4142420 con 0x7f0ca40b4d60 2024-03-25T19:46:00.360 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.360+0000 7f0ca0ff9640 1 -- 172.21.15.44:0/4265609909 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 379855853 0 0) 0x7f0ca41435a0 con 0x7f0ca40b1730 2024-03-25T19:46:00.360 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.360+0000 7f0ca0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:00.360 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.360+0000 7f0ca0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:00.361 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.360+0000 7f0ca0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:00.361 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.360+0000 7f0ca0ff9640 10 monclient(hunting): my global_id is 4691 2024-03-25T19:46:00.361 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.360+0000 7f0ca0ff9640 1 -- 172.21.15.44:0/4265609909 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ca4142970 con 0x7f0ca40b1730 2024-03-25T19:46:00.390 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.390+0000 7f0ca0ff9640 1 -- 172.21.15.44:0/4265609909 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2053251057 0 0) 0x7f0ca4142970 con 0x7f0ca40b1730 2024-03-25T19:46:00.390 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.390+0000 7f0ca0ff9640 1 -- 172.21.15.44:0/4265609909 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ca41435a0 con 0x7f0ca40b1730 2024-03-25T19:46:00.390 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.390+0000 7f0ca0ff9640 1 -- 172.21.15.44:0/4265609909 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f0c94001830 con 0x7f0ca40b1730 2024-03-25T19:46:00.390 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.390+0000 7f0ca0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:00.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.390+0000 7f0ca0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:00.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.390+0000 7f0ca0ff9640 10 monclient(hunting): dump: 2024-03-25T19:46:00.391 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:00.391 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:00.391 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:00.391 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:00.391 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:00.391 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:00.391 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:00.391 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:00.391 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:00.391 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:00.391 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.390+0000 7f0ca0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:00.396 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.395+0000 7f0ca0ff9640 1 -- 172.21.15.44:0/4265609909 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 614249638 0 0) 0x7f0ca4142420 con 0x7f0ca40b4d60 2024-03-25T19:46:00.396 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.396+0000 7f0ca0ff9640 1 -- 172.21.15.44:0/4265609909 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ca4142970 con 0x7f0ca40b4d60 2024-03-25T19:46:00.396 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.396+0000 7f0ca0ff9640 1 -- 172.21.15.44:0/4265609909 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f0c98001950 con 0x7f0ca40b4d60 2024-03-25T19:46:00.396 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.396+0000 7f0ca0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:00.396 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.396+0000 7f0ca0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:00.396 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.396+0000 7f0ca0ff9640 10 monclient(hunting): dump: 2024-03-25T19:46:00.397 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:00.397 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:00.397 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:00.397 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:00.397 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:00.397 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:00.397 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:00.397 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:00.397 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:00.397 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:00.401 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.400+0000 7f0ca0ff9640 1 -- 172.21.15.44:0/4265609909 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2027361651 0 0) 0x7f0c90004eb0 con 0x7f0ca414be90 2024-03-25T19:46:00.401 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.400+0000 7f0ca0ff9640 1 -- 172.21.15.44:0/4265609909 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ca4142420 con 0x7f0ca414be90 2024-03-25T19:46:00.401 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.400+0000 7f0ca0ff9640 1 -- 172.21.15.44:0/4265609909 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f0c9c001f60 con 0x7f0ca414be90 2024-03-25T19:46:00.401 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.400+0000 7f0ca0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:00.401 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.400+0000 7f0ca0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:00.401 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.400+0000 7f0ca0ff9640 10 monclient(hunting): dump: 2024-03-25T19:46:00.401 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:00.401 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:00.401 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:00.402 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:00.402 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:00.402 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:00.402 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:00.402 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:00.402 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:00.402 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:00.404 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.401+0000 7f0ca0ff9640 1 -- 172.21.15.44:0/4265609909 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2748847357 0 0) 0x7f0ca41435a0 con 0x7f0ca40b1730 2024-03-25T19:46:00.404 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.401+0000 7f0ca0ff9640 1 -- 172.21.15.44:0/4265609909 >> v1:172.21.15.111:6789/0 conn(0x7f0ca40b4d60 legacy=0x7f0ca4142e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:00.404 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.401+0000 7f0ca0ff9640 1 -- 172.21.15.44:0/4265609909 >> v1:172.21.15.44:6789/0 conn(0x7f0ca414be90 legacy=0x7f0ca41e1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:00.404 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.401+0000 7f0ca0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:46:00.404 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.401+0000 7f0ca0ff9640 1 monclient: found mon.c 2024-03-25T19:46:00.404 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.401+0000 7f0ca0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:00.404 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.401+0000 7f0ca0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:46:00.404 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.401+0000 7f0ca0ff9640 1 -- 172.21.15.44:0/4265609909 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca4141060 con 0x7f0ca40b1730 2024-03-25T19:46:00.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.401+0000 7f0ca0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:46:00.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.401+0000 7f0ca0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:46:00.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.401+0000 7f0ca0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:00.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.401+0000 7f0caa938640 5 monclient: authenticate success, global_id 4691 2024-03-25T19:46:00.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.401+0000 7f0caa938640 10 monclient: _renew_subs 2024-03-25T19:46:00.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.401+0000 7f0caa938640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:46:00.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.401+0000 7f0caa938640 1 -- 172.21.15.44:0/4265609909 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca4138540 con 0x7f0ca40b1730 2024-03-25T19:46:00.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.401+0000 7f0caa938640 10 client.4691.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:00.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.401+0000 7f0caa938640 10 monclient: _renew_subs 2024-03-25T19:46:00.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.401+0000 7f0caa938640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:46:00.405 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.401+0000 7f0caa938640 1 -- 172.21.15.44:0/4265609909 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca41e3570 con 0x7f0ca40b1730 2024-03-25T19:46:00.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.401+0000 7f0caa938640 1 librados: init done 2024-03-25T19:46:00.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.402+0000 7f0caa938640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:00.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.402+0000 7f0caa938640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:00.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.402+0000 7f0caa938640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:46:00.406 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.402+0000 7f0caa938640 1 -- 172.21.15.44:0/4265609909 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ca41401f0 con 0x7f0ca40b1730 2024-03-25T19:46:00.719 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.719+0000 7f0ca0ff9640 1 -- 172.21.15.44:0/4265609909 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c94001d40 con 0x7f0ca40b1730 2024-03-25T19:46:00.719 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.719+0000 7f0ca0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:00.719 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.719+0000 7f0ca0ff9640 1 -- 172.21.15.44:0/4265609909 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f0c940037f0 con 0x7f0ca40b1730 2024-03-25T19:46:00.719 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.719+0000 7f0ca0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:00.719 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.719+0000 7f0ca0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:00.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.719+0000 7f0ca0ff9640 10 monclient: dump: 2024-03-25T19:46:00.720 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:00.720 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:00.720 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:00.720 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:00.720 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:00.720 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:00.720 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:00.720 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:00.720 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:00.720 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:00.720 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.719+0000 7f0ca1ffb640 4 set_mon_vals no callback set 2024-03-25T19:46:00.721 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.720+0000 7f0ca0ff9640 1 -- 172.21.15.44:0/4265609909 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f0c94019880 con 0x7f0ca40b1730 2024-03-25T19:46:00.721 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.720+0000 7f0ca37fe640 10 monclient: get_auth_request con 0x7f0c9006e0f0 auth_method 0 2024-03-25T19:46:00.721 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.720+0000 7f0ca0ff9640 10 client.4691.objecter ms_handle_connect 0x7f0c9006e0f0 2024-03-25T19:46:00.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.728+0000 7f0ca0ff9640 1 -- 172.21.15.44:0/4265609909 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1275227418 0 0) 0x7f0c94053560 con 0x7f0ca40b1730 2024-03-25T19:46:00.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.728+0000 7f0ca0ff9640 10 client.4691.objecter ms_dispatch 0x7f0ca4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:46:00.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.728+0000 7f0ca0ff9640 3 client.4691.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:46:00.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.728+0000 7f0ca0ff9640 3 client.4691.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:46:00.728 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.728+0000 7f0ca0ff9640 20 client.4691.objecter dump_active .. 0 homeless 2024-03-25T19:46:00.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.728+0000 7f0ca0ff9640 1 -- 172.21.15.44:0/4265609909 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0c9404eea0 con 0x7f0ca40b1730 2024-03-25T19:46:00.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.728+0000 7f0ca0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:00.729 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.728+0000 7f0ca0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:00.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.969+0000 7f0caa938640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:00.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.969+0000 7f0caa938640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:00.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.969+0000 7f0caa938640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:46:00.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.969+0000 7f0caa938640 1 -- 172.21.15.44:0/4265609909 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ca41427d0 con 0x7f0ca40b1730 2024-03-25T19:46:00.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.994+0000 7f0ca0ff9640 1 -- 172.21.15.44:0/4265609909 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2782837909) 0x7f0ca41427d0 con 0x7f0ca40b1730 2024-03-25T19:46:00.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.994+0000 7f0ca0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:00.995 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.994+0000 7f0ca0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:00.995 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:46:00.996 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":11,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:45:30.548832+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:45:30.548832+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:00.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.996+0000 7f0caa938640 10 librados: watch_flush enter 2024-03-25T19:46:00.997 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.996+0000 7f0caa938640 10 librados: watch_flush exit 2024-03-25T19:46:00.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.996+0000 7f0caa938640 20 client.4691.objecter shutdown clearing up homeless session... 2024-03-25T19:46:00.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.996+0000 7f0caa938640 10 client.4691.objecter successfully canceled tick 2024-03-25T19:46:00.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.997+0000 7f0caa938640 1 -- 172.21.15.44:0/4265609909 >> v1:172.21.15.111:6800/3883215793 conn(0x7f0c9006e0f0 legacy=0x7f0c900705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:00.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.997+0000 7f0caa938640 10 monclient: shutdown 2024-03-25T19:46:00.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.997+0000 7f0caa938640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:00.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.997+0000 7f0caa938640 1 -- 172.21.15.44:0/4265609909 >> v1:172.21.15.44:6790/0 conn(0x7f0ca40b1730 legacy=0x7f0ca4140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:00.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.997+0000 7f0caa938640 1 -- 172.21.15.44:0/4265609909 shutdown_connections 2024-03-25T19:46:00.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.997+0000 7f0caa938640 1 -- 172.21.15.44:0/4265609909 >> 172.21.15.44:0/4265609909 conn(0x7f0ca40617a0 msgr2=0x7f0ca40b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:00.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.997+0000 7f0caa938640 1 -- 172.21.15.44:0/4265609909 shutdown_connections 2024-03-25T19:46:00.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.997+0000 7f0caa938640 1 -- 172.21.15.44:0/4265609909 wait complete. 2024-03-25T19:46:00.998 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:00.997+0000 7f0caa938640 1 librados: shutdown 2024-03-25T19:46:01.008 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:05.724 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:10:56.625 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:05.770 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:10:56.649 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:05.771 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:10:56.672 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:05.771 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:10:56.650 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:05.781 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:10:56.660 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:05.817 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:10:56.719 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:05.849 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:10:56.751 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:05.856 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:10:56.757 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:05.885 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:10:56.787 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:07.816 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:07.931 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.931+0000 7f618d6d4640 10 monclient: get_monmap_and_config 2024-03-25T19:46:07.931 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.931+0000 7f618d6d4640 10 monclient: build_initial_monmap 2024-03-25T19:46:07.931 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.931+0000 7f618d6d4640 10 monclient: monmap: 2024-03-25T19:46:07.931 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:46:07.931 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:07.931 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:46:07.932403+0000 2024-03-25T19:46:07.932 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:46:07.932403+0000 2024-03-25T19:46:07.932 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:07.932 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:07.932 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:46:07.932 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:46:07.932 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:46:07.932 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:07.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.932+0000 7f618d6d4640 1 Processor -- start 2024-03-25T19:46:07.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.932+0000 7f618d6d4640 1 -- start start 2024-03-25T19:46:07.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.932+0000 7f618d6d4640 10 monclient: init 2024-03-25T19:46:07.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.933+0000 7f618d6d4640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:07.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.933+0000 7f618d6d4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:07.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.933+0000 7f618d6d4640 10 monclient(hunting): picked mon.noname-a con 0x7f6188070710 addr v1:172.21.15.44:6789/0 2024-03-25T19:46:07.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.933+0000 7f618d6d4640 10 monclient(hunting): picked mon.noname-c con 0x7f6188140c50 addr v1:172.21.15.111:6789/0 2024-03-25T19:46:07.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.933+0000 7f618d6d4640 10 monclient(hunting): picked mon.noname-b con 0x7f618813d520 addr v1:172.21.15.44:6790/0 2024-03-25T19:46:07.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.933+0000 7f618d6d4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6188070f10 con 0x7f6188070710 2024-03-25T19:46:07.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.933+0000 7f618d6d4640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6188072410 con 0x7f6188140c50 2024-03-25T19:46:07.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.933+0000 7f618d6d4640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61880726f0 con 0x7f618813d520 2024-03-25T19:46:07.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.933+0000 7f618d6d4640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:07.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.933+0000 7f618d6d4640 10 monclient(hunting): authenticate will time out at 6032.104371s 2024-03-25T19:46:07.949 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.949+0000 7f6186ffd640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f6188070710 0x7f6188140540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:46598/0 (socket says 172.21.15.44:46598) 2024-03-25T19:46:07.949 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.949+0000 7f6186ffd640 1 -- 172.21.15.44:0/1550299966 learned_addr learned my addr 172.21.15.44:0/1550299966 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:46:07.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.958+0000 7f6185ffb640 1 -- 172.21.15.44:0/1550299966 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813755766 0 0) 0x7f6188070f10 con 0x7f6188070710 2024-03-25T19:46:07.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.959+0000 7f6185ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:07.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.959+0000 7f6185ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:07.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.959+0000 7f6185ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:07.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.959+0000 7f6185ffb640 10 monclient(hunting): my global_id is 4707 2024-03-25T19:46:07.959 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.959+0000 7f6185ffb640 1 -- 172.21.15.44:0/1550299966 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6170003520 con 0x7f6188070710 2024-03-25T19:46:07.960 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.960+0000 7f6185ffb640 1 -- 172.21.15.44:0/1550299966 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 32939214 0 0) 0x7f6188072410 con 0x7f6188140c50 2024-03-25T19:46:07.960 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.960+0000 7f6185ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:07.960 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.960+0000 7f6185ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:07.960 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.960+0000 7f6185ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:07.960 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.960+0000 7f6185ffb640 10 monclient(hunting): my global_id is 4702 2024-03-25T19:46:07.961 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.960+0000 7f6185ffb640 1 -- 172.21.15.44:0/1550299966 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6188070f10 con 0x7f6188140c50 2024-03-25T19:46:07.965 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.965+0000 7f6185ffb640 1 -- 172.21.15.44:0/1550299966 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 597796980 0 0) 0x7f61880726f0 con 0x7f618813d520 2024-03-25T19:46:07.966 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.965+0000 7f6185ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:07.966 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.965+0000 7f6185ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:07.966 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.965+0000 7f6185ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:07.966 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.965+0000 7f6185ffb640 10 monclient(hunting): my global_id is 4694 2024-03-25T19:46:07.966 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.965+0000 7f6185ffb640 1 -- 172.21.15.44:0/1550299966 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6188072410 con 0x7f618813d520 2024-03-25T19:46:07.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.975+0000 7f6185ffb640 1 -- 172.21.15.44:0/1550299966 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3626813106 0 0) 0x7f6170003520 con 0x7f6188070710 2024-03-25T19:46:07.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.975+0000 7f6185ffb640 1 -- 172.21.15.44:0/1550299966 >> v1:172.21.15.44:6790/0 conn(0x7f618813d520 legacy=0x7f618814be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:07.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.975+0000 7f6185ffb640 1 -- 172.21.15.44:0/1550299966 >> v1:172.21.15.111:6789/0 conn(0x7f6188140c50 legacy=0x7f618813ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:07.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.975+0000 7f6185ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:46:07.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.975+0000 7f6185ffb640 1 monclient: found mon.noname-a 2024-03-25T19:46:07.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.975+0000 7f6185ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:07.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.975+0000 7f6185ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.44:6789/0 2024-03-25T19:46:07.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.975+0000 7f6185ffb640 1 -- 172.21.15.44:0/1550299966 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61880729d0 con 0x7f6188070710 2024-03-25T19:46:07.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.975+0000 7f6185ffb640 10 monclient: _finish_auth 0 2024-03-25T19:46:07.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.975+0000 7f6185ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:46:07.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.975+0000 7f6185ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:07.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.975+0000 7f6185ffb640 1 -- 172.21.15.44:0/1550299966 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6178001690 con 0x7f6188070710 2024-03-25T19:46:07.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.975+0000 7f618d6d4640 5 monclient: authenticate success, global_id 4707 2024-03-25T19:46:07.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.975+0000 7f618d6d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:07.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.975+0000 7f6185ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:07.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.975+0000 7f6185ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:07.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.975+0000 7f6185ffb640 10 monclient: dump: 2024-03-25T19:46:07.977 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:07.977 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:07.977 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:07.977 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:07.977 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:07.977 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:07.977 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:07.977 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:07.977 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:07.977 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:07.977 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.975+0000 7f618d6d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:08.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.999+0000 7f6185ffb640 1 -- 172.21.15.44:0/1550299966 <== mon.0 v1:172.21.15.44:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61780017f0 con 0x7f6188070710 2024-03-25T19:46:08.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.999+0000 7f6185ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:08.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.999+0000 7f6185ffb640 1 -- 172.21.15.44:0/1550299966 <== mon.0 v1:172.21.15.44:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6178001b80 con 0x7f6188070710 2024-03-25T19:46:08.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.999+0000 7f6185ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:08.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.999+0000 7f6185ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:08.000 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:07.999+0000 7f6185ffb640 10 monclient: dump: 2024-03-25T19:46:08.000 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:08.001 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:08.001 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:08.001 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:08.001 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:08.001 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:08.001 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:08.001 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:08.001 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:08.001 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:08.001 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.000+0000 7f618d6d4640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:08.001 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.000+0000 7f618d6d4640 4 set_mon_vals no callback set 2024-03-25T19:46:08.004 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.000+0000 7f618d6d4640 10 monclient: shutdown 2024-03-25T19:46:08.004 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.000+0000 7f618d6d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:08.004 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.000+0000 7f618d6d4640 1 -- 172.21.15.44:0/1550299966 >> v1:172.21.15.44:6789/0 conn(0x7f6188070710 legacy=0x7f6188140540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:08.004 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.000+0000 7f618d6d4640 1 -- 172.21.15.44:0/1550299966 shutdown_connections 2024-03-25T19:46:08.004 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.000+0000 7f618d6d4640 1 -- 172.21.15.44:0/1550299966 >> 172.21.15.44:0/1550299966 conn(0x7f61880b5d10 msgr2=0x7f61880b6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:08.004 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.001+0000 7f618d6d4640 1 -- 172.21.15.44:0/1550299966 shutdown_connections 2024-03-25T19:46:08.004 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.001+0000 7f618d6d4640 1 -- 172.21.15.44:0/1550299966 wait complete. 2024-03-25T19:46:08.004 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.001+0000 7f618d6d4640 10 monclient: build_initial_monmap 2024-03-25T19:46:08.004 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.001+0000 7f618d6d4640 10 monclient: monmap: 2024-03-25T19:46:08.004 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:46:08.004 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:08.004 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:46:08.005 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:46:08.005 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:08.005 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:46:08.005 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:46:08.005 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:46:08.005 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:46:08.005 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:08.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.002+0000 7f618d6d4640 1 librados: starting msgr at 2024-03-25T19:46:08.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.002+0000 7f618d6d4640 1 librados: starting objecter 2024-03-25T19:46:08.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.002+0000 7f618d6d4640 1 Processor -- start 2024-03-25T19:46:08.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.002+0000 7f618d6d4640 1 -- start start 2024-03-25T19:46:08.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.002+0000 7f618d6d4640 1 librados: setting wanted keys 2024-03-25T19:46:08.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.002+0000 7f618d6d4640 1 librados: calling monclient init 2024-03-25T19:46:08.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.002+0000 7f618d6d4640 10 monclient: init 2024-03-25T19:46:08.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.003+0000 7f618d6d4640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:08.005 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.003+0000 7f618d6d4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:08.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.003+0000 7f618d6d4640 10 monclient(hunting): picked mon.noname-a con 0x7f6188070710 addr v1:172.21.15.44:6789/0 2024-03-25T19:46:08.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.003+0000 7f618d6d4640 10 monclient(hunting): picked mon.noname-b con 0x7f618813d520 addr v1:172.21.15.111:6789/0 2024-03-25T19:46:08.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.003+0000 7f618d6d4640 10 monclient(hunting): picked mon.noname-c con 0x7f6188140c50 addr v1:172.21.15.44:6790/0 2024-03-25T19:46:08.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.003+0000 7f618d6d4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6188142500 con 0x7f6188070710 2024-03-25T19:46:08.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.003+0000 7f618d6d4640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61880af8b0 con 0x7f618813d520 2024-03-25T19:46:08.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.003+0000 7f618d6d4640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61881d3ed0 con 0x7f6188140c50 2024-03-25T19:46:08.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.003+0000 7f618d6d4640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:08.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.003+0000 7f618d6d4640 10 monclient(hunting): authenticate will time out at 6032.174330s 2024-03-25T19:46:08.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.010+0000 7f61867fc640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f618813d520 0x7f61881d37c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:42566/0 (socket says 172.21.15.44:42566) 2024-03-25T19:46:08.010 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.010+0000 7f61867fc640 1 -- 172.21.15.44:0/2092291655 learned_addr learned my addr 172.21.15.44:0/2092291655 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:46:08.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.011+0000 7f616f7fe640 10 client.?.objecter ms_handle_connect 0x7f618813d520 2024-03-25T19:46:08.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.011+0000 7f616f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:08.021 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.020+0000 7f616f7fe640 1 -- 172.21.15.44:0/2092291655 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741980942 0 0) 0x7f61880af8b0 con 0x7f618813d520 2024-03-25T19:46:08.021 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.020+0000 7f616f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:08.021 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.020+0000 7f616f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:08.021 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.020+0000 7f616f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:08.021 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.020+0000 7f616f7fe640 10 monclient(hunting): my global_id is 4705 2024-03-25T19:46:08.021 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.020+0000 7f616f7fe640 1 -- 172.21.15.44:0/2092291655 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6174004eb0 con 0x7f618813d520 2024-03-25T19:46:08.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.026+0000 7f616f7fe640 10 client.?.objecter ms_handle_connect 0x7f6188140c50 2024-03-25T19:46:08.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.026+0000 7f616f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:08.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.033+0000 7f616f7fe640 10 client.?.objecter ms_handle_connect 0x7f6188070710 2024-03-25T19:46:08.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.033+0000 7f616f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:08.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.041+0000 7f616f7fe640 1 -- 172.21.15.44:0/2092291655 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3600815771 0 0) 0x7f6174004eb0 con 0x7f618813d520 2024-03-25T19:46:08.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.041+0000 7f616f7fe640 1 -- 172.21.15.44:0/2092291655 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61880af8b0 con 0x7f618813d520 2024-03-25T19:46:08.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.041+0000 7f616f7fe640 1 -- 172.21.15.44:0/2092291655 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f617c001a00 con 0x7f618813d520 2024-03-25T19:46:08.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.041+0000 7f616f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:08.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.041+0000 7f616f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:08.042 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.041+0000 7f616f7fe640 10 monclient(hunting): dump: 2024-03-25T19:46:08.042 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:08.042 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:08.042 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:08.042 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:08.043 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:08.043 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:08.043 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:08.043 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:08.043 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:08.043 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:08.043 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.041+0000 7f616f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:08.043 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.043+0000 7f616f7fe640 1 -- 172.21.15.44:0/2092291655 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3270194779 0 0) 0x7f6188142500 con 0x7f6188070710 2024-03-25T19:46:08.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.043+0000 7f616f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:08.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.043+0000 7f616f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:08.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.043+0000 7f616f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:08.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.043+0000 7f616f7fe640 10 monclient(hunting): my global_id is 4710 2024-03-25T19:46:08.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.043+0000 7f616f7fe640 1 -- 172.21.15.44:0/2092291655 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6174004eb0 con 0x7f6188070710 2024-03-25T19:46:08.044 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.044+0000 7f616f7fe640 1 -- 172.21.15.44:0/2092291655 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3999861828 0 0) 0x7f61881d3ed0 con 0x7f6188140c50 2024-03-25T19:46:08.045 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.044+0000 7f616f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:08.045 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.044+0000 7f616f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:08.045 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.044+0000 7f616f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:08.045 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.044+0000 7f616f7fe640 10 monclient(hunting): my global_id is 4697 2024-03-25T19:46:08.045 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.044+0000 7f616f7fe640 1 -- 172.21.15.44:0/2092291655 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6188142500 con 0x7f6188140c50 2024-03-25T19:46:08.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.049+0000 7f616f7fe640 1 -- 172.21.15.44:0/2092291655 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2013421179 0 0) 0x7f61880af8b0 con 0x7f618813d520 2024-03-25T19:46:08.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.049+0000 7f616f7fe640 1 -- 172.21.15.44:0/2092291655 >> v1:172.21.15.44:6790/0 conn(0x7f6188140c50 legacy=0x7f61880ae090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:08.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.049+0000 7f616f7fe640 1 -- 172.21.15.44:0/2092291655 >> v1:172.21.15.44:6789/0 conn(0x7f6188070710 legacy=0x7f6188143180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:08.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.049+0000 7f616f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:46:08.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.049+0000 7f616f7fe640 1 monclient: found mon.b 2024-03-25T19:46:08.050 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.049+0000 7f616f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:08.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.049+0000 7f616f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:46:08.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.049+0000 7f616f7fe640 1 -- 172.21.15.44:0/2092291655 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61880aeda0 con 0x7f618813d520 2024-03-25T19:46:08.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.049+0000 7f616f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:46:08.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.049+0000 7f616f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:46:08.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.049+0000 7f616f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:08.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.049+0000 7f618d6d4640 5 monclient: authenticate success, global_id 4705 2024-03-25T19:46:08.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.049+0000 7f618d6d4640 10 monclient: _renew_subs 2024-03-25T19:46:08.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.050+0000 7f618d6d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:46:08.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.050+0000 7f618d6d4640 1 -- 172.21.15.44:0/2092291655 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61880aef90 con 0x7f618813d520 2024-03-25T19:46:08.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.050+0000 7f618d6d4640 10 client.4705.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:08.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.050+0000 7f618d6d4640 10 monclient: _renew_subs 2024-03-25T19:46:08.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.050+0000 7f618d6d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:46:08.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.050+0000 7f618d6d4640 1 -- 172.21.15.44:0/2092291655 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6188142b70 con 0x7f618813d520 2024-03-25T19:46:08.051 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.050+0000 7f618d6d4640 1 librados: init done 2024-03-25T19:46:08.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.050+0000 7f618d6d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:08.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.050+0000 7f618d6d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:08.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.050+0000 7f618d6d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:46:08.052 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.050+0000 7f618d6d4640 1 -- 172.21.15.44:0/2092291655 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61881410b0 con 0x7f618813d520 2024-03-25T19:46:08.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.069+0000 7f616f7fe640 1 -- 172.21.15.44:0/2092291655 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f617c001ca0 con 0x7f618813d520 2024-03-25T19:46:08.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.069+0000 7f616f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:08.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.069+0000 7f616f7fe640 1 -- 172.21.15.44:0/2092291655 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f617c0038c0 con 0x7f618813d520 2024-03-25T19:46:08.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.069+0000 7f616f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:08.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.069+0000 7f616f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:08.070 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.069+0000 7f616f7fe640 10 monclient: dump: 2024-03-25T19:46:08.070 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:08.070 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:08.070 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:08.070 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:08.070 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:08.071 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:08.071 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:08.071 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:08.071 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:08.071 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:08.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.069+0000 7f61857fa640 4 set_mon_vals no callback set 2024-03-25T19:46:08.071 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.070+0000 7f616f7fe640 1 -- 172.21.15.44:0/2092291655 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f617c003ae0 con 0x7f618813d520 2024-03-25T19:46:08.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.070+0000 7f6186ffd640 10 monclient: get_auth_request con 0x7f617406de10 auth_method 0 2024-03-25T19:46:08.072 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.071+0000 7f616f7fe640 10 client.4705.objecter ms_handle_connect 0x7f617406de10 2024-03-25T19:46:08.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.329+0000 7f616f7fe640 1 -- 172.21.15.44:0/2092291655 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1275227418 0 0) 0x7f617c0546d0 con 0x7f618813d520 2024-03-25T19:46:08.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.329+0000 7f616f7fe640 10 client.4705.objecter ms_dispatch 0x7f6188000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:46:08.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.329+0000 7f616f7fe640 3 client.4705.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:46:08.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.329+0000 7f616f7fe640 3 client.4705.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:46:08.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.329+0000 7f616f7fe640 20 client.4705.objecter dump_active .. 0 homeless 2024-03-25T19:46:08.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.329+0000 7f616f7fe640 1 -- 172.21.15.44:0/2092291655 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f617c0501e0 con 0x7f618813d520 2024-03-25T19:46:08.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.330+0000 7f616f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:08.330 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.330+0000 7f616f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:08.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.548+0000 7f618d6d4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:08.548 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.548+0000 7f618d6d4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:08.549 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.548+0000 7f618d6d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:46:08.549 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.548+0000 7f618d6d4640 1 -- 172.21.15.44:0/2092291655 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6188142260 con 0x7f618813d520 2024-03-25T19:46:08.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.592+0000 7f616f7fe640 1 -- 172.21.15.44:0/2092291655 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2782837909) 0x7f617c052110 con 0x7f618813d520 2024-03-25T19:46:08.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.592+0000 7f616f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:08.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.592+0000 7f616f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:08.593 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:46:08.593 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":11,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:45:30.548832+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:45:30.548832+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:08.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.594+0000 7f618d6d4640 10 librados: watch_flush enter 2024-03-25T19:46:08.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.594+0000 7f618d6d4640 10 librados: watch_flush exit 2024-03-25T19:46:08.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.594+0000 7f618d6d4640 20 client.4705.objecter shutdown clearing up homeless session... 2024-03-25T19:46:08.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.594+0000 7f618d6d4640 10 client.4705.objecter successfully canceled tick 2024-03-25T19:46:08.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.594+0000 7f618d6d4640 1 -- 172.21.15.44:0/2092291655 >> v1:172.21.15.111:6800/3883215793 conn(0x7f617406de10 legacy=0x7f61740702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:08.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.594+0000 7f618d6d4640 10 monclient: shutdown 2024-03-25T19:46:08.595 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.594+0000 7f618d6d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:08.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.594+0000 7f618d6d4640 1 -- 172.21.15.44:0/2092291655 >> v1:172.21.15.111:6789/0 conn(0x7f618813d520 legacy=0x7f61881d37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:08.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.594+0000 7f618d6d4640 1 -- 172.21.15.44:0/2092291655 shutdown_connections 2024-03-25T19:46:08.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.595+0000 7f618d6d4640 1 -- 172.21.15.44:0/2092291655 >> 172.21.15.44:0/2092291655 conn(0x7f61880b5d10 msgr2=0x7f6188074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:08.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.595+0000 7f618d6d4640 1 -- 172.21.15.44:0/2092291655 shutdown_connections 2024-03-25T19:46:08.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.595+0000 7f618d6d4640 1 -- 172.21.15.44:0/2092291655 wait complete. 2024-03-25T19:46:08.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:08.595+0000 7f618d6d4640 1 librados: shutdown 2024-03-25T19:46:08.607 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:15.414 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:15.520 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.520+0000 7fa9ea73d640 10 monclient: get_monmap_and_config 2024-03-25T19:46:15.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.520+0000 7fa9ea73d640 10 monclient: build_initial_monmap 2024-03-25T19:46:15.521 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.520+0000 7fa9ea73d640 10 monclient: monmap: 2024-03-25T19:46:15.521 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:46:15.521 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:15.521 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:46:15.521723+0000 2024-03-25T19:46:15.521 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:46:15.521723+0000 2024-03-25T19:46:15.521 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:15.521 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:15.521 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:46:15.521 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:46:15.521 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:46:15.521 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:15.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.521+0000 7fa9ea73d640 1 Processor -- start 2024-03-25T19:46:15.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.521+0000 7fa9ea73d640 1 -- start start 2024-03-25T19:46:15.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.521+0000 7fa9ea73d640 10 monclient: init 2024-03-25T19:46:15.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.522+0000 7fa9ea73d640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:15.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.522+0000 7fa9ea73d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:46:15.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.522+0000 7fa9ea73d640 10 monclient(hunting): picked mon.noname-b con 0x7fa9e40b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T19:46:15.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.522+0000 7fa9ea73d640 10 monclient(hunting): picked mon.noname-a con 0x7fa9e40b1730 addr v1:172.21.15.44:6789/0 2024-03-25T19:46:15.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.522+0000 7fa9ea73d640 10 monclient(hunting): picked mon.noname-c con 0x7fa9e414be90 addr v1:172.21.15.111:6789/0 2024-03-25T19:46:15.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.522+0000 7fa9ea73d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e40af940 con 0x7fa9e40b1730 2024-03-25T19:46:15.523 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.522+0000 7fa9ea73d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e40b0a70 con 0x7fa9e414be90 2024-03-25T19:46:15.524 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.522+0000 7fa9ea73d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e40b0d50 con 0x7fa9e40b4d60 2024-03-25T19:46:15.524 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.522+0000 7fa9ea73d640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:15.524 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.522+0000 7fa9ea73d640 10 monclient(hunting): authenticate will time out at 6039.692957s 2024-03-25T19:46:15.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.532+0000 7fa9e8cda640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fa9e414be90 0x7fa9e414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:52214/0 (socket says 172.21.15.44:52214) 2024-03-25T19:46:15.532 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.532+0000 7fa9e8cda640 1 -- 172.21.15.44:0/3175824968 learned_addr learned my addr 172.21.15.44:0/3175824968 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:46:15.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.542+0000 7fa9e2ffd640 1 -- 172.21.15.44:0/3175824968 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 451386876 0 0) 0x7fa9e40b0a70 con 0x7fa9e414be90 2024-03-25T19:46:15.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.542+0000 7fa9e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:15.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.542+0000 7fa9e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:15.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.542+0000 7fa9e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:15.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.542+0000 7fa9e2ffd640 10 monclient(hunting): my global_id is 4708 2024-03-25T19:46:15.542 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.542+0000 7fa9e2ffd640 1 -- 172.21.15.44:0/3175824968 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9cc003520 con 0x7fa9e414be90 2024-03-25T19:46:15.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.544+0000 7fa9e2ffd640 1 -- 172.21.15.44:0/3175824968 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048465790 0 0) 0x7fa9e40af940 con 0x7fa9e40b1730 2024-03-25T19:46:15.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.544+0000 7fa9e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:15.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.544+0000 7fa9e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:15.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.544+0000 7fa9e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:15.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.544+0000 7fa9e2ffd640 10 monclient(hunting): my global_id is 4713 2024-03-25T19:46:15.544 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.544+0000 7fa9e2ffd640 1 -- 172.21.15.44:0/3175824968 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9e40b0a70 con 0x7fa9e40b1730 2024-03-25T19:46:15.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.550+0000 7fa9e2ffd640 1 -- 172.21.15.44:0/3175824968 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 973870832 0 0) 0x7fa9cc003520 con 0x7fa9e414be90 2024-03-25T19:46:15.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.551+0000 7fa9e2ffd640 1 -- 172.21.15.44:0/3175824968 >> v1:172.21.15.44:6790/0 conn(0x7fa9e40b4d60 legacy=0x7fa9e413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:15.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.551+0000 7fa9e2ffd640 1 -- 172.21.15.44:0/3175824968 >> v1:172.21.15.44:6789/0 conn(0x7fa9e40b1730 legacy=0x7fa9e40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:15.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.551+0000 7fa9e2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:15.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.551+0000 7fa9e2ffd640 1 monclient: found mon.noname-c 2024-03-25T19:46:15.551 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.551+0000 7fa9e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:15.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.551+0000 7fa9e2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:46:15.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.551+0000 7fa9e2ffd640 1 -- 172.21.15.44:0/3175824968 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e40b1030 con 0x7fa9e414be90 2024-03-25T19:46:15.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.551+0000 7fa9e2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:46:15.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.551+0000 7fa9e2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:15.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.551+0000 7fa9e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:15.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.551+0000 7fa9e2ffd640 1 -- 172.21.15.44:0/3175824968 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fa9dc001a10 con 0x7fa9e414be90 2024-03-25T19:46:15.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.551+0000 7fa9e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:15.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.551+0000 7fa9e2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:15.552 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.551+0000 7fa9e2ffd640 10 monclient: dump: 2024-03-25T19:46:15.552 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:15.552 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:15.552 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:15.552 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:15.553 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:15.553 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:15.553 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:15.553 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:15.553 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:15.553 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:15.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.551+0000 7fa9ea73d640 5 monclient: authenticate success, global_id 4708 2024-03-25T19:46:15.553 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.551+0000 7fa9ea73d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:15.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.567+0000 7fa9e2ffd640 1 -- 172.21.15.44:0/3175824968 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9dc001b70 con 0x7fa9e414be90 2024-03-25T19:46:15.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.567+0000 7fa9e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:15.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.567+0000 7fa9e2ffd640 1 -- 172.21.15.44:0/3175824968 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fa9dc001f80 con 0x7fa9e414be90 2024-03-25T19:46:15.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.567+0000 7fa9ea73d640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:15.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.567+0000 7fa9ea73d640 4 set_mon_vals no callback set 2024-03-25T19:46:15.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.567+0000 7fa9e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:15.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.567+0000 7fa9e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:15.569 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.567+0000 7fa9e2ffd640 10 monclient: dump: 2024-03-25T19:46:15.569 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:15.569 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:15.570 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:15.570 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:15.570 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:15.570 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:15.570 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:15.570 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:15.570 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:15.570 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:15.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.568+0000 7fa9ea73d640 10 monclient: shutdown 2024-03-25T19:46:15.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.568+0000 7fa9ea73d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:15.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.568+0000 7fa9ea73d640 1 -- 172.21.15.44:0/3175824968 >> v1:172.21.15.111:6789/0 conn(0x7fa9e414be90 legacy=0x7fa9e414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:15.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.568+0000 7fa9ea73d640 1 -- 172.21.15.44:0/3175824968 shutdown_connections 2024-03-25T19:46:15.570 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.568+0000 7fa9ea73d640 1 -- 172.21.15.44:0/3175824968 >> 172.21.15.44:0/3175824968 conn(0x7fa9e40617a0 msgr2=0x7fa9e4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:15.571 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.568+0000 7fa9ea73d640 1 -- 172.21.15.44:0/3175824968 shutdown_connections 2024-03-25T19:46:15.571 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.568+0000 7fa9ea73d640 1 -- 172.21.15.44:0/3175824968 wait complete. 2024-03-25T19:46:15.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.569+0000 7fa9ea73d640 10 monclient: build_initial_monmap 2024-03-25T19:46:15.572 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.569+0000 7fa9ea73d640 10 monclient: monmap: 2024-03-25T19:46:15.572 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:46:15.572 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:15.572 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:46:15.572 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:46:15.572 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:15.572 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:46:15.572 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:46:15.573 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:46:15.573 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:46:15.573 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:15.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.569+0000 7fa9ea73d640 1 librados: starting msgr at 2024-03-25T19:46:15.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.569+0000 7fa9ea73d640 1 librados: starting objecter 2024-03-25T19:46:15.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.569+0000 7fa9ea73d640 1 Processor -- start 2024-03-25T19:46:15.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.570+0000 7fa9ea73d640 1 -- start start 2024-03-25T19:46:15.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.570+0000 7fa9ea73d640 1 librados: setting wanted keys 2024-03-25T19:46:15.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.570+0000 7fa9ea73d640 1 librados: calling monclient init 2024-03-25T19:46:15.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.570+0000 7fa9ea73d640 10 monclient: init 2024-03-25T19:46:15.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.570+0000 7fa9ea73d640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:15.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.570+0000 7fa9ea73d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:46:15.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.570+0000 7fa9ea73d640 10 monclient(hunting): picked mon.noname-b con 0x7fa9e40b1730 addr v1:172.21.15.111:6789/0 2024-03-25T19:46:15.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.571+0000 7fa9ea73d640 10 monclient(hunting): picked mon.noname-c con 0x7fa9e40b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T19:46:15.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.571+0000 7fa9ea73d640 10 monclient(hunting): picked mon.noname-a con 0x7fa9e414be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:46:15.573 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.571+0000 7fa9ea73d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e4143900 con 0x7fa9e414be90 2024-03-25T19:46:15.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.571+0000 7fa9ea73d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e413b820 con 0x7fa9e40b1730 2024-03-25T19:46:15.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.571+0000 7fa9ea73d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e413ba70 con 0x7fa9e40b4d60 2024-03-25T19:46:15.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.571+0000 7fa9ea73d640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:15.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.571+0000 7fa9ea73d640 10 monclient(hunting): authenticate will time out at 6039.741903s 2024-03-25T19:46:15.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.585+0000 7fa9e37fe640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7fa9e40b4d60 0x7fa9e41429f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:57246/0 (socket says 172.21.15.44:57246) 2024-03-25T19:46:15.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.585+0000 7fa9e37fe640 1 -- 172.21.15.44:0/243540164 learned_addr learned my addr 172.21.15.44:0/243540164 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:46:15.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.586+0000 7fa9e0ff9640 10 client.?.objecter ms_handle_connect 0x7fa9e40b4d60 2024-03-25T19:46:15.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.586+0000 7fa9e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:15.591 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.591+0000 7fa9e0ff9640 10 client.?.objecter ms_handle_connect 0x7fa9e40b1730 2024-03-25T19:46:15.592 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.591+0000 7fa9e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:15.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.596+0000 7fa9e0ff9640 10 client.?.objecter ms_handle_connect 0x7fa9e414be90 2024-03-25T19:46:15.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.596+0000 7fa9e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:15.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.608+0000 7fa9e0ff9640 1 -- 172.21.15.44:0/243540164 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021326002 0 0) 0x7fa9e413b820 con 0x7fa9e40b1730 2024-03-25T19:46:15.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.609+0000 7fa9e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:15.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.609+0000 7fa9e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:15.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.609+0000 7fa9e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:15.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.609+0000 7fa9e0ff9640 10 monclient(hunting): my global_id is 4711 2024-03-25T19:46:15.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.609+0000 7fa9e0ff9640 1 -- 172.21.15.44:0/243540164 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9d0004eb0 con 0x7fa9e40b1730 2024-03-25T19:46:15.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.616+0000 7fa9e0ff9640 1 -- 172.21.15.44:0/243540164 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3793324335 0 0) 0x7fa9e413ba70 con 0x7fa9e40b4d60 2024-03-25T19:46:15.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.616+0000 7fa9e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:15.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.616+0000 7fa9e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:15.616 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.616+0000 7fa9e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:15.617 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.616+0000 7fa9e0ff9640 10 monclient(hunting): my global_id is 4703 2024-03-25T19:46:15.617 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.616+0000 7fa9e0ff9640 1 -- 172.21.15.44:0/243540164 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9e413b820 con 0x7fa9e40b4d60 2024-03-25T19:46:15.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.619+0000 7fa9e0ff9640 1 -- 172.21.15.44:0/243540164 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230161265 0 0) 0x7fa9e4143900 con 0x7fa9e414be90 2024-03-25T19:46:15.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.619+0000 7fa9e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:15.619 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.619+0000 7fa9e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:15.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.619+0000 7fa9e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:15.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.619+0000 7fa9e0ff9640 10 monclient(hunting): my global_id is 4716 2024-03-25T19:46:15.620 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.619+0000 7fa9e0ff9640 1 -- 172.21.15.44:0/243540164 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9e413ba70 con 0x7fa9e414be90 2024-03-25T19:46:15.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.635+0000 7fa9e0ff9640 1 -- 172.21.15.44:0/243540164 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 602241966 0 0) 0x7fa9e413ba70 con 0x7fa9e414be90 2024-03-25T19:46:15.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.636+0000 7fa9e0ff9640 1 -- 172.21.15.44:0/243540164 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9e4143900 con 0x7fa9e414be90 2024-03-25T19:46:15.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.636+0000 7fa9e0ff9640 1 -- 172.21.15.44:0/243540164 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fa9dc001e80 con 0x7fa9e414be90 2024-03-25T19:46:15.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.636+0000 7fa9e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:15.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.636+0000 7fa9e0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:15.636 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.636+0000 7fa9e0ff9640 10 monclient(hunting): dump: 2024-03-25T19:46:15.636 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:15.636 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:15.636 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:15.636 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:15.637 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:15.637 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:15.637 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:15.637 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:15.637 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:15.637 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:15.637 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.636+0000 7fa9e0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:15.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.647+0000 7fa9e0ff9640 1 -- 172.21.15.44:0/243540164 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4284734677 0 0) 0x7fa9d0004eb0 con 0x7fa9e40b1730 2024-03-25T19:46:15.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.647+0000 7fa9e0ff9640 1 -- 172.21.15.44:0/243540164 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9e413ba70 con 0x7fa9e40b1730 2024-03-25T19:46:15.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.647+0000 7fa9e0ff9640 1 -- 172.21.15.44:0/243540164 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fa9d40018c0 con 0x7fa9e40b1730 2024-03-25T19:46:15.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.647+0000 7fa9e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:15.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.647+0000 7fa9e0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:15.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.647+0000 7fa9e0ff9640 10 monclient(hunting): dump: 2024-03-25T19:46:15.648 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:15.648 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:15.648 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:15.648 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:15.648 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:15.648 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:15.648 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:15.648 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:15.648 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:15.648 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:15.650 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.650+0000 7fa9e0ff9640 1 -- 172.21.15.44:0/243540164 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2704048482 0 0) 0x7fa9e413b820 con 0x7fa9e40b4d60 2024-03-25T19:46:15.650 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.650+0000 7fa9e0ff9640 1 -- 172.21.15.44:0/243540164 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9d0004eb0 con 0x7fa9e40b4d60 2024-03-25T19:46:15.650 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.650+0000 7fa9e0ff9640 1 -- 172.21.15.44:0/243540164 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fa9d80019f0 con 0x7fa9e40b4d60 2024-03-25T19:46:15.650 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.650+0000 7fa9e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:15.650 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.650+0000 7fa9e0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:15.651 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.650+0000 7fa9e0ff9640 10 monclient(hunting): dump: 2024-03-25T19:46:15.651 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:15.651 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:15.651 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:15.651 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:15.651 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:15.651 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:15.651 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:15.651 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:15.651 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:15.651 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:15.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.668+0000 7fa9e0ff9640 1 -- 172.21.15.44:0/243540164 <== mon.0 v1:172.21.15.44:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4180764896 0 0) 0x7fa9e4143900 con 0x7fa9e414be90 2024-03-25T19:46:15.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.668+0000 7fa9e0ff9640 1 -- 172.21.15.44:0/243540164 >> v1:172.21.15.44:6790/0 conn(0x7fa9e40b4d60 legacy=0x7fa9e41429f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:15.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.668+0000 7fa9e0ff9640 1 -- 172.21.15.44:0/243540164 >> v1:172.21.15.111:6789/0 conn(0x7fa9e40b1730 legacy=0x7fa9e413a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:15.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.668+0000 7fa9e0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:46:15.669 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.668+0000 7fa9e0ff9640 1 monclient: found mon.a 2024-03-25T19:46:15.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.668+0000 7fa9e0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:15.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.668+0000 7fa9e0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:46:15.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.668+0000 7fa9e0ff9640 1 -- 172.21.15.44:0/243540164 --> v1:172.21.15.44:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e413bd50 con 0x7fa9e414be90 2024-03-25T19:46:15.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.668+0000 7fa9e0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:46:15.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.668+0000 7fa9e0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:46:15.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.668+0000 7fa9e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:15.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.668+0000 7fa9ea73d640 5 monclient: authenticate success, global_id 4716 2024-03-25T19:46:15.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.668+0000 7fa9ea73d640 10 monclient: _renew_subs 2024-03-25T19:46:15.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.668+0000 7fa9ea73d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:46:15.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.668+0000 7fa9ea73d640 1 -- 172.21.15.44:0/243540164 --> v1:172.21.15.44:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e413a770 con 0x7fa9e414be90 2024-03-25T19:46:15.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.668+0000 7fa9ea73d640 10 client.4716.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:15.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.668+0000 7fa9ea73d640 10 monclient: _renew_subs 2024-03-25T19:46:15.670 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.668+0000 7fa9ea73d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:46:15.671 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.668+0000 7fa9ea73d640 1 -- 172.21.15.44:0/243540164 --> v1:172.21.15.44:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e41432e0 con 0x7fa9e414be90 2024-03-25T19:46:15.671 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.668+0000 7fa9ea73d640 1 librados: init done 2024-03-25T19:46:15.671 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.669+0000 7fa9ea73d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:15.671 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.669+0000 7fa9ea73d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:15.671 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.669+0000 7fa9ea73d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:46:15.671 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:15.669+0000 7fa9ea73d640 1 -- 172.21.15.44:0/243540164 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e41d79e0 con 0x7fa9e414be90 2024-03-25T19:46:16.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.031+0000 7fa9e0ff9640 1 -- 172.21.15.44:0/243540164 <== mon.0 v1:172.21.15.44:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9dc002020 con 0x7fa9e414be90 2024-03-25T19:46:16.031 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.031+0000 7fa9e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:16.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.031+0000 7fa9e0ff9640 1 -- 172.21.15.44:0/243540164 <== mon.0 v1:172.21.15.44:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fa9dc004fb0 con 0x7fa9e414be90 2024-03-25T19:46:16.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.031+0000 7fa9e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:16.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.031+0000 7fa9e0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:16.032 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.031+0000 7fa9e0ff9640 10 monclient: dump: 2024-03-25T19:46:16.032 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:16.032 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:16.032 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:16.032 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:16.032 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:16.032 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:16.032 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:16.032 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:16.032 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:16.032 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:16.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.031+0000 7fa9e27fc640 4 set_mon_vals no callback set 2024-03-25T19:46:16.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.032+0000 7fa9e0ff9640 1 -- 172.21.15.44:0/243540164 <== mon.0 v1:172.21.15.44:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7fa9dc01af80 con 0x7fa9e414be90 2024-03-25T19:46:16.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.033+0000 7fa9e3fff640 10 monclient: get_auth_request con 0x7fa9d006e060 auth_method 0 2024-03-25T19:46:16.033 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.033+0000 7fa9e0ff9640 10 client.4716.objecter ms_handle_connect 0x7fa9d006e060 2024-03-25T19:46:16.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.038+0000 7fa9e0ff9640 1 -- 172.21.15.44:0/243540164 <== mon.0 v1:172.21.15.44:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1275227418 0 0) 0x7fa9dc054a70 con 0x7fa9e414be90 2024-03-25T19:46:16.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.038+0000 7fa9e0ff9640 10 client.4716.objecter ms_dispatch 0x7fa9e4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:46:16.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.038+0000 7fa9e0ff9640 3 client.4716.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:46:16.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.038+0000 7fa9e0ff9640 3 client.4716.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:46:16.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.039+0000 7fa9e0ff9640 20 client.4716.objecter dump_active .. 0 homeless 2024-03-25T19:46:16.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.039+0000 7fa9e0ff9640 1 -- 172.21.15.44:0/243540164 <== mon.0 v1:172.21.15.44:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa9dc050580 con 0x7fa9e414be90 2024-03-25T19:46:16.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.039+0000 7fa9e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:16.039 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.039+0000 7fa9e0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:16.287 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.287+0000 7fa9ea73d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:16.287 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.287+0000 7fa9ea73d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:16.287 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.287+0000 7fa9ea73d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.44:6789/0 2024-03-25T19:46:16.288 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.287+0000 7fa9ea73d640 1 -- 172.21.15.44:0/243540164 --> v1:172.21.15.44:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9e40627b0 con 0x7fa9e414be90 2024-03-25T19:46:16.331 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.331+0000 7fa9e0ff9640 1 -- 172.21.15.44:0/243540164 <== mon.0 v1:172.21.15.44:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2782837909) 0x7fa9dc0524b0 con 0x7fa9e414be90 2024-03-25T19:46:16.331 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.331+0000 7fa9e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:16.331 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.331+0000 7fa9e0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:16.332 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:46:16.332 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":11,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:45:30.548832+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:45:30.548832+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:16.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.333+0000 7fa9ea73d640 10 librados: watch_flush enter 2024-03-25T19:46:16.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.333+0000 7fa9ea73d640 10 librados: watch_flush exit 2024-03-25T19:46:16.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.333+0000 7fa9ea73d640 20 client.4716.objecter shutdown clearing up homeless session... 2024-03-25T19:46:16.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.333+0000 7fa9ea73d640 10 client.4716.objecter successfully canceled tick 2024-03-25T19:46:16.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.333+0000 7fa9ea73d640 1 -- 172.21.15.44:0/243540164 >> v1:172.21.15.111:6800/3883215793 conn(0x7fa9d006e060 legacy=0x7fa9d0070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:16.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.333+0000 7fa9ea73d640 10 monclient: shutdown 2024-03-25T19:46:16.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.333+0000 7fa9ea73d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:16.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.333+0000 7fa9ea73d640 1 -- 172.21.15.44:0/243540164 >> v1:172.21.15.44:6789/0 conn(0x7fa9e414be90 legacy=0x7fa9e41d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:16.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.333+0000 7fa9ea73d640 1 -- 172.21.15.44:0/243540164 shutdown_connections 2024-03-25T19:46:16.335 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.333+0000 7fa9ea73d640 1 -- 172.21.15.44:0/243540164 >> 172.21.15.44:0/243540164 conn(0x7fa9e40617a0 msgr2=0x7fa9e40b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:16.335 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.333+0000 7fa9ea73d640 1 -- 172.21.15.44:0/243540164 shutdown_connections 2024-03-25T19:46:16.335 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.333+0000 7fa9ea73d640 1 -- 172.21.15.44:0/243540164 wait complete. 2024-03-25T19:46:16.335 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:16.333+0000 7fa9ea73d640 1 librados: shutdown 2024-03-25T19:46:16.346 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:23.152 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:23.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.266+0000 7fc96953b640 10 monclient: get_monmap_and_config 2024-03-25T19:46:23.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.266+0000 7fc96953b640 10 monclient: build_initial_monmap 2024-03-25T19:46:23.266 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.266+0000 7fc96953b640 10 monclient: monmap: 2024-03-25T19:46:23.267 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:46:23.267 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:23.267 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:46:23.267458+0000 2024-03-25T19:46:23.267 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:46:23.267458+0000 2024-03-25T19:46:23.267 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:23.267 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:23.267 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:46:23.267 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:46:23.267 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:46:23.267 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:23.268 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.267+0000 7fc96953b640 1 Processor -- start 2024-03-25T19:46:23.268 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.267+0000 7fc96953b640 1 -- start start 2024-03-25T19:46:23.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.267+0000 7fc96953b640 10 monclient: init 2024-03-25T19:46:23.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.268+0000 7fc96953b640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:23.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.268+0000 7fc96953b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:46:23.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.268+0000 7fc96953b640 10 monclient(hunting): picked mon.noname-c con 0x7fc9640705b0 addr v1:172.21.15.111:6789/0 2024-03-25T19:46:23.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.268+0000 7fc96953b640 10 monclient(hunting): picked mon.noname-a con 0x7fc9641402d0 addr v1:172.21.15.44:6789/0 2024-03-25T19:46:23.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.268+0000 7fc96953b640 10 monclient(hunting): picked mon.noname-b con 0x7fc964149ae0 addr v1:172.21.15.44:6790/0 2024-03-25T19:46:23.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.268+0000 7fc96953b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9640718c0 con 0x7fc9641402d0 2024-03-25T19:46:23.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.268+0000 7fc96953b640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9640714c0 con 0x7fc9640705b0 2024-03-25T19:46:23.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.268+0000 7fc96953b640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc964072b50 con 0x7fc964149ae0 2024-03-25T19:46:23.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.268+0000 7fc96953b640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:23.269 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.268+0000 7fc96953b640 10 monclient(hunting): authenticate will time out at 6047.439103s 2024-03-25T19:46:23.277 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.276+0000 7fc962ffd640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc9640705b0 0x7fc96413d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:56912/0 (socket says 172.21.15.44:56912) 2024-03-25T19:46:23.277 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.276+0000 7fc962ffd640 1 -- 172.21.15.44:0/1817069187 learned_addr learned my addr 172.21.15.44:0/1817069187 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:46:23.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.284+0000 7fc9627fc640 1 -- 172.21.15.44:0/1817069187 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788822404 0 0) 0x7fc9640714c0 con 0x7fc9640705b0 2024-03-25T19:46:23.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.285+0000 7fc9627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:23.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.285+0000 7fc9627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:23.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.285+0000 7fc9627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:23.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.285+0000 7fc9627fc640 10 monclient(hunting): my global_id is 4714 2024-03-25T19:46:23.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.285+0000 7fc9627fc640 1 -- 172.21.15.44:0/1817069187 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc950003520 con 0x7fc9640705b0 2024-03-25T19:46:23.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.296+0000 7fc9627fc640 1 -- 172.21.15.44:0/1817069187 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3962111526 0 0) 0x7fc950003520 con 0x7fc9640705b0 2024-03-25T19:46:23.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.296+0000 7fc9627fc640 1 -- 172.21.15.44:0/1817069187 >> v1:172.21.15.44:6790/0 conn(0x7fc964149ae0 legacy=0x7fc96414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:23.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.296+0000 7fc9627fc640 1 -- 172.21.15.44:0/1817069187 >> v1:172.21.15.44:6789/0 conn(0x7fc9641402d0 legacy=0x7fc964140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:46:23.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.296+0000 7fc9627fc640 10 monclient: _finish_hunting 0 2024-03-25T19:46:23.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.296+0000 7fc9627fc640 1 monclient: found mon.noname-c 2024-03-25T19:46:23.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.296+0000 7fc9627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:23.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.296+0000 7fc9627fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:46:23.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.296+0000 7fc9627fc640 1 -- 172.21.15.44:0/1817069187 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc96413dd10 con 0x7fc9640705b0 2024-03-25T19:46:23.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.296+0000 7fc9627fc640 10 monclient: _finish_auth 0 2024-03-25T19:46:23.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.296+0000 7fc9627fc640 10 monclient: _check_auth_tickets 2024-03-25T19:46:23.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.296+0000 7fc9627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:23.297 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.296+0000 7fc9627fc640 1 -- 172.21.15.44:0/1817069187 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fc954001690 con 0x7fc9640705b0 2024-03-25T19:46:23.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.296+0000 7fc96953b640 5 monclient: authenticate success, global_id 4714 2024-03-25T19:46:23.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.296+0000 7fc96953b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:23.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.296+0000 7fc9627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:23.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.296+0000 7fc9627fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:23.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.296+0000 7fc9627fc640 10 monclient: dump: 2024-03-25T19:46:23.298 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:23.298 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:23.298 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:23.298 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:23.298 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:23.298 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:23.298 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:23.298 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:23.298 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:23.299 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:23.299 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.296+0000 7fc96953b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:23.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.305+0000 7fc9627fc640 1 -- 172.21.15.44:0/1817069187 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc954001860 con 0x7fc9640705b0 2024-03-25T19:46:23.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.305+0000 7fc9627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:23.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.305+0000 7fc9627fc640 1 -- 172.21.15.44:0/1817069187 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fc954001cb0 con 0x7fc9640705b0 2024-03-25T19:46:23.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.305+0000 7fc96953b640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:23.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.305+0000 7fc9627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:23.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.305+0000 7fc9627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:23.306 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.305+0000 7fc96953b640 4 set_mon_vals no callback set 2024-03-25T19:46:23.306 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.305+0000 7fc9627fc640 10 monclient: dump: 2024-03-25T19:46:23.306 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:23.306 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:23.306 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:23.306 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:23.306 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:23.306 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:23.306 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:23.306 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:23.306 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:23.306 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:23.307 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.306+0000 7fc96953b640 10 monclient: shutdown 2024-03-25T19:46:23.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.306+0000 7fc96953b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:23.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.306+0000 7fc96953b640 1 -- 172.21.15.44:0/1817069187 >> v1:172.21.15.111:6789/0 conn(0x7fc9640705b0 legacy=0x7fc96413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:23.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.306+0000 7fc96953b640 1 -- 172.21.15.44:0/1817069187 shutdown_connections 2024-03-25T19:46:23.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.306+0000 7fc96953b640 1 -- 172.21.15.44:0/1817069187 >> 172.21.15.44:0/1817069187 conn(0x7fc9640b5d10 msgr2=0x7fc9640b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:23.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.306+0000 7fc96953b640 1 -- 172.21.15.44:0/1817069187 shutdown_connections 2024-03-25T19:46:23.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.306+0000 7fc96953b640 1 -- 172.21.15.44:0/1817069187 wait complete. 2024-03-25T19:46:23.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.307+0000 7fc96953b640 10 monclient: build_initial_monmap 2024-03-25T19:46:23.308 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.307+0000 7fc96953b640 10 monclient: monmap: 2024-03-25T19:46:23.308 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:46:23.308 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:23.308 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:46:23.308 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:46:23.309 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:23.309 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:46:23.309 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:46:23.309 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:46:23.309 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:46:23.309 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:23.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.307+0000 7fc96953b640 1 librados: starting msgr at 2024-03-25T19:46:23.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.307+0000 7fc96953b640 1 librados: starting objecter 2024-03-25T19:46:23.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.307+0000 7fc96953b640 1 Processor -- start 2024-03-25T19:46:23.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.307+0000 7fc96953b640 1 -- start start 2024-03-25T19:46:23.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.307+0000 7fc96953b640 1 librados: setting wanted keys 2024-03-25T19:46:23.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.307+0000 7fc96953b640 1 librados: calling monclient init 2024-03-25T19:46:23.309 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.307+0000 7fc96953b640 10 monclient: init 2024-03-25T19:46:23.310 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.308+0000 7fc96953b640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:23.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.308+0000 7fc96953b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:46:23.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.308+0000 7fc96953b640 10 monclient(hunting): picked mon.noname-b con 0x7fc9640705b0 addr v1:172.21.15.111:6789/0 2024-03-25T19:46:23.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.308+0000 7fc96953b640 10 monclient(hunting): picked mon.noname-c con 0x7fc9641402d0 addr v1:172.21.15.44:6790/0 2024-03-25T19:46:23.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.308+0000 7fc96953b640 10 monclient(hunting): picked mon.noname-a con 0x7fc964149ae0 addr v1:172.21.15.44:6789/0 2024-03-25T19:46:23.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.308+0000 7fc96953b640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc964146fc0 con 0x7fc964149ae0 2024-03-25T19:46:23.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.308+0000 7fc96953b640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9641472a0 con 0x7fc9640705b0 2024-03-25T19:46:23.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.308+0000 7fc96953b640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9641dee70 con 0x7fc9641402d0 2024-03-25T19:46:23.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.308+0000 7fc96953b640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:23.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.308+0000 7fc96953b640 10 monclient(hunting): authenticate will time out at 6047.479651s 2024-03-25T19:46:23.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.310+0000 7fc962ffd640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc9640705b0 0x7fc964145d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:56934/0 (socket says 172.21.15.44:56934) 2024-03-25T19:46:23.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.310+0000 7fc962ffd640 1 -- 172.21.15.44:0/4257582697 learned_addr learned my addr 172.21.15.44:0/4257582697 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:46:23.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.310+0000 7fc95b7fe640 10 client.?.objecter ms_handle_connect 0x7fc9640705b0 2024-03-25T19:46:23.311 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.310+0000 7fc95b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:23.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.311+0000 7fc95b7fe640 10 client.?.objecter ms_handle_connect 0x7fc964149ae0 2024-03-25T19:46:23.312 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.311+0000 7fc95b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:23.317 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.317+0000 7fc95b7fe640 10 client.?.objecter ms_handle_connect 0x7fc9641402d0 2024-03-25T19:46:23.317 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.317+0000 7fc95b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:23.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.321+0000 7fc95b7fe640 1 -- 172.21.15.44:0/4257582697 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345399325 0 0) 0x7fc9641472a0 con 0x7fc9640705b0 2024-03-25T19:46:23.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.321+0000 7fc95b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:23.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.321+0000 7fc95b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:23.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.321+0000 7fc95b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:23.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.321+0000 7fc95b7fe640 10 monclient(hunting): my global_id is 4717 2024-03-25T19:46:23.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.321+0000 7fc95b7fe640 1 -- 172.21.15.44:0/4257582697 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc944004eb0 con 0x7fc9640705b0 2024-03-25T19:46:23.323 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.323+0000 7fc95b7fe640 1 -- 172.21.15.44:0/4257582697 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909217694 0 0) 0x7fc9641dee70 con 0x7fc9641402d0 2024-03-25T19:46:23.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.323+0000 7fc95b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:23.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.323+0000 7fc95b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:23.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.323+0000 7fc95b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:23.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.323+0000 7fc95b7fe640 10 monclient(hunting): my global_id is 4709 2024-03-25T19:46:23.324 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.323+0000 7fc95b7fe640 1 -- 172.21.15.44:0/4257582697 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9641472a0 con 0x7fc9641402d0 2024-03-25T19:46:23.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.325+0000 7fc95b7fe640 1 -- 172.21.15.44:0/4257582697 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1954623047 0 0) 0x7fc964146fc0 con 0x7fc964149ae0 2024-03-25T19:46:23.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.325+0000 7fc95b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:23.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.325+0000 7fc95b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:23.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.325+0000 7fc95b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:23.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.325+0000 7fc95b7fe640 10 monclient(hunting): my global_id is 4719 2024-03-25T19:46:23.326 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.325+0000 7fc95b7fe640 1 -- 172.21.15.44:0/4257582697 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9641dee70 con 0x7fc964149ae0 2024-03-25T19:46:23.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.334+0000 7fc95b7fe640 1 -- 172.21.15.44:0/4257582697 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1913522997 0 0) 0x7fc9641472a0 con 0x7fc9641402d0 2024-03-25T19:46:23.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.334+0000 7fc95b7fe640 1 -- 172.21.15.44:0/4257582697 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc964146fc0 con 0x7fc9641402d0 2024-03-25T19:46:23.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.334+0000 7fc95b7fe640 1 -- 172.21.15.44:0/4257582697 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fc94c001810 con 0x7fc9641402d0 2024-03-25T19:46:23.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.334+0000 7fc95b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:23.334 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.334+0000 7fc95b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:23.335 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.334+0000 7fc95b7fe640 10 monclient(hunting): dump: 2024-03-25T19:46:23.335 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:23.335 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:23.335 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:23.335 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:23.335 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:23.335 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:23.335 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:23.335 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:23.335 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:23.335 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:23.335 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.334+0000 7fc95b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:23.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.339+0000 7fc95b7fe640 1 -- 172.21.15.44:0/4257582697 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2714169155 0 0) 0x7fc944004eb0 con 0x7fc9640705b0 2024-03-25T19:46:23.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.339+0000 7fc95b7fe640 1 -- 172.21.15.44:0/4257582697 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9641472a0 con 0x7fc9640705b0 2024-03-25T19:46:23.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.339+0000 7fc95b7fe640 1 -- 172.21.15.44:0/4257582697 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fc954001b70 con 0x7fc9640705b0 2024-03-25T19:46:23.339 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.339+0000 7fc95b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:23.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.339+0000 7fc95b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:23.340 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.339+0000 7fc95b7fe640 10 monclient(hunting): dump: 2024-03-25T19:46:23.340 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:23.340 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:23.340 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:23.340 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:23.340 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:23.340 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:23.340 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:23.340 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:23.340 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:23.340 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:23.347 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.347+0000 7fc95b7fe640 1 -- 172.21.15.44:0/4257582697 <== mon.0 v1:172.21.15.44:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3415143921 0 0) 0x7fc9641dee70 con 0x7fc964149ae0 2024-03-25T19:46:23.347 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.347+0000 7fc95b7fe640 1 -- 172.21.15.44:0/4257582697 --> v1:172.21.15.44:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc944004eb0 con 0x7fc964149ae0 2024-03-25T19:46:23.347 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.347+0000 7fc95b7fe640 1 -- 172.21.15.44:0/4257582697 <== mon.0 v1:172.21.15.44:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fc95c001a90 con 0x7fc964149ae0 2024-03-25T19:46:23.347 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.347+0000 7fc95b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:23.347 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.347+0000 7fc95b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:23.347 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.347+0000 7fc95b7fe640 10 monclient(hunting): dump: 2024-03-25T19:46:23.347 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:23.348 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:23.348 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:23.348 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:23.348 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:23.348 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:23.348 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:23.348 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:23.348 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:23.348 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:23.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.352+0000 7fc95b7fe640 1 -- 172.21.15.44:0/4257582697 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3895596790 0 0) 0x7fc964146fc0 con 0x7fc9641402d0 2024-03-25T19:46:23.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.353+0000 7fc95b7fe640 1 -- 172.21.15.44:0/4257582697 >> v1:172.21.15.111:6789/0 conn(0x7fc9640705b0 legacy=0x7fc964145d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:23.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.353+0000 7fc95b7fe640 1 -- 172.21.15.44:0/4257582697 >> v1:172.21.15.44:6789/0 conn(0x7fc964149ae0 legacy=0x7fc964148d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:23.353 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.353+0000 7fc95b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:46:23.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.353+0000 7fc95b7fe640 1 monclient: found mon.c 2024-03-25T19:46:23.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.353+0000 7fc95b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:23.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.353+0000 7fc95b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:46:23.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.353+0000 7fc95b7fe640 1 -- 172.21.15.44:0/4257582697 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9641425a0 con 0x7fc9641402d0 2024-03-25T19:46:23.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.353+0000 7fc95b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:46:23.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.353+0000 7fc95b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:46:23.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.353+0000 7fc95b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:23.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.353+0000 7fc96953b640 5 monclient: authenticate success, global_id 4709 2024-03-25T19:46:23.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.353+0000 7fc96953b640 10 monclient: _renew_subs 2024-03-25T19:46:23.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.353+0000 7fc96953b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:46:23.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.353+0000 7fc96953b640 1 -- 172.21.15.44:0/4257582697 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9641dfaa0 con 0x7fc9641402d0 2024-03-25T19:46:23.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.353+0000 7fc96953b640 10 client.4709.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:23.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.353+0000 7fc96953b640 10 monclient: _renew_subs 2024-03-25T19:46:23.354 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.353+0000 7fc96953b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:46:23.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.353+0000 7fc96953b640 1 -- 172.21.15.44:0/4257582697 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc964149660 con 0x7fc9641402d0 2024-03-25T19:46:23.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.353+0000 7fc96953b640 1 librados: init done 2024-03-25T19:46:23.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.354+0000 7fc96953b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:23.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.354+0000 7fc96953b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:23.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.354+0000 7fc96953b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:46:23.355 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.354+0000 7fc96953b640 1 -- 172.21.15.44:0/4257582697 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc964149950 con 0x7fc9641402d0 2024-03-25T19:46:23.675 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.675+0000 7fc95b7fe640 1 -- 172.21.15.44:0/4257582697 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc94c001d00 con 0x7fc9641402d0 2024-03-25T19:46:23.675 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.675+0000 7fc95b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:23.675 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.675+0000 7fc95b7fe640 1 -- 172.21.15.44:0/4257582697 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fc94c003790 con 0x7fc9641402d0 2024-03-25T19:46:23.675 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.675+0000 7fc95b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:23.675 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.675+0000 7fc95b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:23.675 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.675+0000 7fc95b7fe640 10 monclient: dump: 2024-03-25T19:46:23.675 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:23.676 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:23.676 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:23.676 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:23.676 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:23.676 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:23.676 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:23.676 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:23.676 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:23.676 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:23.676 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.675+0000 7fc961ffb640 4 set_mon_vals no callback set 2024-03-25T19:46:23.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.675+0000 7fc95b7fe640 1 -- 172.21.15.44:0/4257582697 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7fc94c0197a0 con 0x7fc9641402d0 2024-03-25T19:46:23.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.676+0000 7fc962ffd640 10 monclient: get_auth_request con 0x7fc94406e0b0 auth_method 0 2024-03-25T19:46:23.677 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.676+0000 7fc95b7fe640 10 client.4709.objecter ms_handle_connect 0x7fc94406e0b0 2024-03-25T19:46:23.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.681+0000 7fc95b7fe640 1 -- 172.21.15.44:0/4257582697 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1275227418 0 0) 0x7fc94c0534b0 con 0x7fc9641402d0 2024-03-25T19:46:23.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.681+0000 7fc95b7fe640 10 client.4709.objecter ms_dispatch 0x7fc964000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:46:23.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.681+0000 7fc95b7fe640 3 client.4709.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:46:23.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.681+0000 7fc95b7fe640 3 client.4709.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:46:23.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.681+0000 7fc95b7fe640 20 client.4709.objecter dump_active .. 0 homeless 2024-03-25T19:46:23.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.681+0000 7fc95b7fe640 1 -- 172.21.15.44:0/4257582697 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc94c04edf0 con 0x7fc9641402d0 2024-03-25T19:46:23.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.681+0000 7fc95b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:23.681 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.681+0000 7fc95b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:23.928 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.928+0000 7fc96953b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:23.928 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.928+0000 7fc96953b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:23.929 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.928+0000 7fc96953b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:46:23.929 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.928+0000 7fc96953b640 1 -- 172.21.15.44:0/4257582697 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9641df000 con 0x7fc9641402d0 2024-03-25T19:46:23.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.948+0000 7fc95b7fe640 1 -- 172.21.15.44:0/4257582697 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2782837909) 0x7fc94c050ef0 con 0x7fc9641402d0 2024-03-25T19:46:23.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.948+0000 7fc95b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:23.948 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.948+0000 7fc95b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:23.949 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:46:23.949 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":11,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:45:30.548832+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:45:30.548832+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:23.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.950+0000 7fc96953b640 10 librados: watch_flush enter 2024-03-25T19:46:23.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.950+0000 7fc96953b640 10 librados: watch_flush exit 2024-03-25T19:46:23.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.950+0000 7fc96953b640 20 client.4709.objecter shutdown clearing up homeless session... 2024-03-25T19:46:23.951 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.950+0000 7fc96953b640 10 client.4709.objecter successfully canceled tick 2024-03-25T19:46:23.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.950+0000 7fc96953b640 1 -- 172.21.15.44:0/4257582697 >> v1:172.21.15.111:6800/3883215793 conn(0x7fc94406e0b0 legacy=0x7fc944070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:23.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.950+0000 7fc96953b640 10 monclient: shutdown 2024-03-25T19:46:23.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.950+0000 7fc96953b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:23.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.950+0000 7fc96953b640 1 -- 172.21.15.44:0/4257582697 >> v1:172.21.15.44:6790/0 conn(0x7fc9641402d0 legacy=0x7fc9641df390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:23.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.951+0000 7fc96953b640 1 -- 172.21.15.44:0/4257582697 shutdown_connections 2024-03-25T19:46:23.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.951+0000 7fc96953b640 1 -- 172.21.15.44:0/4257582697 >> 172.21.15.44:0/4257582697 conn(0x7fc9640b5d10 msgr2=0x7fc9640b4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:23.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.951+0000 7fc96953b640 1 -- 172.21.15.44:0/4257582697 shutdown_connections 2024-03-25T19:46:23.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.951+0000 7fc96953b640 1 -- 172.21.15.44:0/4257582697 wait complete. 2024-03-25T19:46:23.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:23.951+0000 7fc96953b640 1 librados: shutdown 2024-03-25T19:46:23.964 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:26.111 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:11:17.012 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:26.118 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:11:17.020 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:26.150 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:11:17.051 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:26.165 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:11:17.045 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:26.172 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:11:17.052 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:26.197 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:11:17.098 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:26.199 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:11:17.079 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:26.205 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:11:17.107 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:26.235 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:11:17.137 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:30.770 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:30.882 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.883+0000 7f5ebf245640 10 monclient: get_monmap_and_config 2024-03-25T19:46:30.882 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.883+0000 7f5ebf245640 10 monclient: build_initial_monmap 2024-03-25T19:46:30.883 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.883+0000 7f5ebf245640 10 monclient: monmap: 2024-03-25T19:46:30.883 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:46:30.883 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:30.883 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:46:30.883358+0000 2024-03-25T19:46:30.883 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:46:30.883358+0000 2024-03-25T19:46:30.883 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:30.883 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:30.883 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:46:30.883 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:46:30.883 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:46:30.883 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:30.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.884+0000 7f5ebf245640 1 Processor -- start 2024-03-25T19:46:30.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.884+0000 7f5ebf245640 1 -- start start 2024-03-25T19:46:30.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.884+0000 7f5ebf245640 10 monclient: init 2024-03-25T19:46:30.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.885+0000 7f5ebf245640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:30.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.885+0000 7f5ebf245640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:30.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.885+0000 7f5ebf245640 10 monclient(hunting): picked mon.noname-b con 0x7f5eb80a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T19:46:30.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.885+0000 7f5ebf245640 10 monclient(hunting): picked mon.noname-c con 0x7f5eb813f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:46:30.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.885+0000 7f5ebf245640 10 monclient(hunting): picked mon.noname-a con 0x7f5eb81430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:46:30.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.885+0000 7f5ebf245640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb8142520 con 0x7f5eb81430c0 2024-03-25T19:46:30.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.885+0000 7f5ebf245640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb80b28a0 con 0x7f5eb813f710 2024-03-25T19:46:30.885 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.885+0000 7f5ebf245640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb80b2a30 con 0x7f5eb80a98f0 2024-03-25T19:46:30.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.885+0000 7f5ebf245640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:30.886 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.885+0000 7f5ebf245640 10 monclient(hunting): authenticate will time out at 6055.055061s 2024-03-25T19:46:30.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.894+0000 7f5ebd7e2640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7f5eb81430c0 0x7f5eb814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:35752/0 (socket says 172.21.15.44:35752) 2024-03-25T19:46:30.893 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.894+0000 7f5ebd7e2640 1 -- 172.21.15.44:0/2441231684 learned_addr learned my addr 172.21.15.44:0/2441231684 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:46:30.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.913+0000 7f5eb77fe640 1 -- 172.21.15.44:0/2441231684 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284935713 0 0) 0x7f5eb80b2a30 con 0x7f5eb80a98f0 2024-03-25T19:46:30.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.913+0000 7f5eb77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:30.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.913+0000 7f5eb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:30.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.913+0000 7f5eb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:30.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.913+0000 7f5eb77fe640 10 monclient(hunting): my global_id is 4712 2024-03-25T19:46:30.912 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.913+0000 7f5eb77fe640 1 -- 172.21.15.44:0/2441231684 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ea8003520 con 0x7f5eb80a98f0 2024-03-25T19:46:30.916 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.917+0000 7f5eb77fe640 1 -- 172.21.15.44:0/2441231684 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3879054658 0 0) 0x7f5eb80b28a0 con 0x7f5eb813f710 2024-03-25T19:46:30.916 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.917+0000 7f5eb77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:30.916 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.917+0000 7f5eb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:30.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.917+0000 7f5eb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:30.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.917+0000 7f5eb77fe640 10 monclient(hunting): my global_id is 4720 2024-03-25T19:46:30.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.917+0000 7f5eb77fe640 1 -- 172.21.15.44:0/2441231684 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb80b2a30 con 0x7f5eb813f710 2024-03-25T19:46:30.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.918+0000 7f5eb77fe640 1 -- 172.21.15.44:0/2441231684 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727918354 0 0) 0x7f5eb8142520 con 0x7f5eb81430c0 2024-03-25T19:46:30.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.918+0000 7f5eb77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:30.917 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.918+0000 7f5eb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:30.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.918+0000 7f5eb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:30.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.918+0000 7f5eb77fe640 10 monclient(hunting): my global_id is 4722 2024-03-25T19:46:30.918 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.918+0000 7f5eb77fe640 1 -- 172.21.15.44:0/2441231684 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb80b28a0 con 0x7f5eb81430c0 2024-03-25T19:46:30.926 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.927+0000 7f5eb77fe640 1 -- 172.21.15.44:0/2441231684 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2552046077 0 0) 0x7f5eb80b2a30 con 0x7f5eb813f710 2024-03-25T19:46:30.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.927+0000 7f5eb77fe640 1 -- 172.21.15.44:0/2441231684 >> v1:172.21.15.44:6790/0 conn(0x7f5eb80a98f0 legacy=0x7f5eb80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:30.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.927+0000 7f5eb77fe640 1 -- 172.21.15.44:0/2441231684 >> v1:172.21.15.44:6789/0 conn(0x7f5eb81430c0 legacy=0x7f5eb814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:30.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.927+0000 7f5eb77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:46:30.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.927+0000 7f5eb77fe640 1 monclient: found mon.noname-c 2024-03-25T19:46:30.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.927+0000 7f5eb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:30.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.927+0000 7f5eb77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:46:30.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.927+0000 7f5eb77fe640 1 -- 172.21.15.44:0/2441231684 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb80b44c0 con 0x7f5eb813f710 2024-03-25T19:46:30.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.927+0000 7f5eb77fe640 10 monclient: _finish_auth 0 2024-03-25T19:46:30.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.927+0000 7f5eb77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:46:30.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.927+0000 7f5eb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:30.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.927+0000 7f5ebf245640 5 monclient: authenticate success, global_id 4720 2024-03-25T19:46:30.927 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.927+0000 7f5ebf245640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:30.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.934+0000 7f5eb77fe640 1 -- 172.21.15.44:0/2441231684 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f5ea4001630 con 0x7f5eb813f710 2024-03-25T19:46:30.933 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.934+0000 7f5eb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:30.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.934+0000 7f5eb77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:30.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.934+0000 7f5eb77fe640 10 monclient: dump: 2024-03-25T19:46:30.934 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:30.934 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:30.934 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:30.934 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:30.934 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:30.934 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:30.934 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:30.934 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:30.934 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:30.934 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:30.934 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.934+0000 7f5ebf245640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:30.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.935+0000 7f5eb77fe640 1 -- 172.21.15.44:0/2441231684 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ea4001880 con 0x7f5eb813f710 2024-03-25T19:46:30.935 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.935+0000 7f5eb77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:30.936 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.935+0000 7f5eb77fe640 1 -- 172.21.15.44:0/2441231684 <== mon.1 v1:172.21.15.111:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f5ea4001c10 con 0x7f5eb813f710 2024-03-25T19:46:30.936 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.935+0000 7f5ebf245640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:30.936 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.935+0000 7f5ebf245640 4 set_mon_vals no callback set 2024-03-25T19:46:30.936 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.935+0000 7f5eb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:30.936 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.935+0000 7f5eb77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:30.936 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.935+0000 7f5eb77fe640 10 monclient: dump: 2024-03-25T19:46:30.936 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:30.936 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:30.936 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:30.936 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:30.936 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:30.936 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:30.936 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:30.937 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:30.937 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:30.937 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:30.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.936+0000 7f5ebf245640 10 monclient: shutdown 2024-03-25T19:46:30.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.936+0000 7f5ebf245640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:30.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.936+0000 7f5ebf245640 1 -- 172.21.15.44:0/2441231684 >> v1:172.21.15.111:6789/0 conn(0x7f5eb813f710 legacy=0x7f5eb81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:30.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.936+0000 7f5ebf245640 1 -- 172.21.15.44:0/2441231684 shutdown_connections 2024-03-25T19:46:30.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.936+0000 7f5ebf245640 1 -- 172.21.15.44:0/2441231684 >> 172.21.15.44:0/2441231684 conn(0x7f5eb80b7ea0 msgr2=0x7f5eb80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:30.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.936+0000 7f5ebf245640 1 -- 172.21.15.44:0/2441231684 shutdown_connections 2024-03-25T19:46:30.937 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.936+0000 7f5ebf245640 1 -- 172.21.15.44:0/2441231684 wait complete. 2024-03-25T19:46:30.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.937+0000 7f5ebf245640 10 monclient: build_initial_monmap 2024-03-25T19:46:30.938 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.937+0000 7f5ebf245640 10 monclient: monmap: 2024-03-25T19:46:30.939 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:46:30.939 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:30.939 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:46:30.939 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:46:30.939 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:30.939 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:46:30.939 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:46:30.939 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:46:30.939 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:46:30.939 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:30.939 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.937+0000 7f5ebf245640 1 librados: starting msgr at 2024-03-25T19:46:30.939 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.937+0000 7f5ebf245640 1 librados: starting objecter 2024-03-25T19:46:30.939 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.937+0000 7f5ebf245640 1 Processor -- start 2024-03-25T19:46:30.939 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.937+0000 7f5ebf245640 1 -- start start 2024-03-25T19:46:30.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.937+0000 7f5ebf245640 1 librados: setting wanted keys 2024-03-25T19:46:30.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.937+0000 7f5ebf245640 1 librados: calling monclient init 2024-03-25T19:46:30.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.937+0000 7f5ebf245640 10 monclient: init 2024-03-25T19:46:30.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.938+0000 7f5ebf245640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:30.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.938+0000 7f5ebf245640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:46:30.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.938+0000 7f5ebf245640 10 monclient(hunting): picked mon.noname-c con 0x7f5eb80a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T19:46:30.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.938+0000 7f5ebf245640 10 monclient(hunting): picked mon.noname-a con 0x7f5eb813f710 addr v1:172.21.15.44:6789/0 2024-03-25T19:46:30.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.938+0000 7f5ebf245640 10 monclient(hunting): picked mon.noname-b con 0x7f5eb81430c0 addr v1:172.21.15.111:6789/0 2024-03-25T19:46:30.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.938+0000 7f5ebf245640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb8136c60 con 0x7f5eb813f710 2024-03-25T19:46:30.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.938+0000 7f5ebf245640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb8135970 con 0x7f5eb81430c0 2024-03-25T19:46:30.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.938+0000 7f5ebf245640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb8136ed0 con 0x7f5eb80a98f0 2024-03-25T19:46:30.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.938+0000 7f5ebf245640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:30.940 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.938+0000 7f5ebf245640 10 monclient(hunting): authenticate will time out at 6055.108567s 2024-03-25T19:46:30.955 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.956+0000 7f5ebd7e2640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f5eb81430c0 0x7f5eb81e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:52452/0 (socket says 172.21.15.44:52452) 2024-03-25T19:46:30.955 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.956+0000 7f5ebd7e2640 1 -- 172.21.15.44:0/1305480252 learned_addr learned my addr 172.21.15.44:0/1305480252 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:46:30.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.956+0000 7f5eb57fa640 10 client.?.objecter ms_handle_connect 0x7f5eb81430c0 2024-03-25T19:46:30.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.956+0000 7f5eb57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:30.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.959+0000 7f5eb57fa640 10 client.?.objecter ms_handle_connect 0x7f5eb813f710 2024-03-25T19:46:30.958 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.959+0000 7f5eb57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:30.966 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.967+0000 7f5eb57fa640 10 client.?.objecter ms_handle_connect 0x7f5eb80a98f0 2024-03-25T19:46:30.966 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.967+0000 7f5eb57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:30.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.980+0000 7f5eb57fa640 1 -- 172.21.15.44:0/1305480252 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317382676 0 0) 0x7f5eb8135970 con 0x7f5eb81430c0 2024-03-25T19:46:30.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.980+0000 7f5eb57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:30.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.980+0000 7f5eb57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:30.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.980+0000 7f5eb57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:30.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.980+0000 7f5eb57fa640 10 monclient(hunting): my global_id is 4723 2024-03-25T19:46:30.980 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.980+0000 7f5eb57fa640 1 -- 172.21.15.44:0/1305480252 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e9c004eb0 con 0x7f5eb81430c0 2024-03-25T19:46:30.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.984+0000 7f5eb57fa640 1 -- 172.21.15.44:0/1305480252 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092811061 0 0) 0x7f5eb8136ed0 con 0x7f5eb80a98f0 2024-03-25T19:46:30.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.984+0000 7f5eb57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:30.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.984+0000 7f5eb57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:30.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.984+0000 7f5eb57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:30.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.984+0000 7f5eb57fa640 10 monclient(hunting): my global_id is 4715 2024-03-25T19:46:30.983 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.984+0000 7f5eb57fa640 1 -- 172.21.15.44:0/1305480252 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb8135970 con 0x7f5eb80a98f0 2024-03-25T19:46:30.992 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.993+0000 7f5eb57fa640 1 -- 172.21.15.44:0/1305480252 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345668473 0 0) 0x7f5eb8136c60 con 0x7f5eb813f710 2024-03-25T19:46:30.992 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.993+0000 7f5eb57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:30.993 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.993+0000 7f5eb57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:30.993 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.993+0000 7f5eb57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:30.993 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.993+0000 7f5eb57fa640 10 monclient(hunting): my global_id is 4725 2024-03-25T19:46:30.993 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:30.993+0000 7f5eb57fa640 1 -- 172.21.15.44:0/1305480252 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb8136ed0 con 0x7f5eb813f710 2024-03-25T19:46:31.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.002+0000 7f5eb57fa640 1 -- 172.21.15.44:0/1305480252 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3379298549 0 0) 0x7f5e9c004eb0 con 0x7f5eb81430c0 2024-03-25T19:46:31.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.003+0000 7f5eb57fa640 1 -- 172.21.15.44:0/1305480252 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5eb8136c60 con 0x7f5eb81430c0 2024-03-25T19:46:31.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.003+0000 7f5eb57fa640 1 -- 172.21.15.44:0/1305480252 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f5eb0001bd0 con 0x7f5eb81430c0 2024-03-25T19:46:31.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.003+0000 7f5eb57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:31.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.003+0000 7f5eb57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:31.002 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.003+0000 7f5eb57fa640 10 monclient(hunting): dump: 2024-03-25T19:46:31.002 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:31.002 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:31.002 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:31.002 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:31.002 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:31.003 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:31.003 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:31.003 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:31.003 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:31.003 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:31.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.003+0000 7f5eb57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:31.004 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.004+0000 7f5eb57fa640 1 -- 172.21.15.44:0/1305480252 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1303934337 0 0) 0x7f5eb8135970 con 0x7f5eb80a98f0 2024-03-25T19:46:31.004 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.004+0000 7f5eb57fa640 1 -- 172.21.15.44:0/1305480252 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e9c004eb0 con 0x7f5eb80a98f0 2024-03-25T19:46:31.004 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.004+0000 7f5eb57fa640 1 -- 172.21.15.44:0/1305480252 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f5eac001950 con 0x7f5eb80a98f0 2024-03-25T19:46:31.004 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.004+0000 7f5eb57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:31.004 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.004+0000 7f5eb57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:31.004 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.004+0000 7f5eb57fa640 10 monclient(hunting): dump: 2024-03-25T19:46:31.004 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:31.004 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:31.004 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:31.004 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:31.004 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:31.004 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:31.004 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:31.005 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:31.005 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:31.005 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:31.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.011+0000 7f5eb57fa640 1 -- 172.21.15.44:0/1305480252 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1943584845 0 0) 0x7f5e9c004eb0 con 0x7f5eb80a98f0 2024-03-25T19:46:31.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.011+0000 7f5eb57fa640 1 -- 172.21.15.44:0/1305480252 >> v1:172.21.15.111:6789/0 conn(0x7f5eb81430c0 legacy=0x7f5eb81e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:31.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.011+0000 7f5eb57fa640 1 -- 172.21.15.44:0/1305480252 >> v1:172.21.15.44:6789/0 conn(0x7f5eb813f710 legacy=0x7f5eb81378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:31.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.011+0000 7f5eb57fa640 10 monclient: _finish_hunting 0 2024-03-25T19:46:31.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.011+0000 7f5eb57fa640 1 monclient: found mon.c 2024-03-25T19:46:31.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.011+0000 7f5eb57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:31.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.011+0000 7f5eb57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:46:31.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.011+0000 7f5eb57fa640 1 -- 172.21.15.44:0/1305480252 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb81371b0 con 0x7f5eb80a98f0 2024-03-25T19:46:31.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.011+0000 7f5eb57fa640 10 monclient: _finish_auth 0 2024-03-25T19:46:31.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.011+0000 7f5eb57fa640 10 monclient: _check_auth_tickets 2024-03-25T19:46:31.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.011+0000 7f5eb57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:31.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.011+0000 7f5ebf245640 5 monclient: authenticate success, global_id 4715 2024-03-25T19:46:31.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.011+0000 7f5ebf245640 10 monclient: _renew_subs 2024-03-25T19:46:31.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.011+0000 7f5ebf245640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:46:31.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.011+0000 7f5ebf245640 1 -- 172.21.15.44:0/1305480252 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb8137490 con 0x7f5eb80a98f0 2024-03-25T19:46:31.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.011+0000 7f5ebf245640 10 client.4715.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:31.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.011+0000 7f5ebf245640 10 monclient: _renew_subs 2024-03-25T19:46:31.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.011+0000 7f5ebf245640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:46:31.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.011+0000 7f5ebf245640 1 -- 172.21.15.44:0/1305480252 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb8135270 con 0x7f5eb80a98f0 2024-03-25T19:46:31.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.011+0000 7f5ebf245640 1 librados: init done 2024-03-25T19:46:31.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.012+0000 7f5ebf245640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:31.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.012+0000 7f5ebf245640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:31.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.012+0000 7f5ebf245640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:46:31.013 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.012+0000 7f5ebf245640 1 -- 172.21.15.44:0/1305480252 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb80627b0 con 0x7f5eb80a98f0 2024-03-25T19:46:31.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.342+0000 7f5eb57fa640 1 -- 172.21.15.44:0/1305480252 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5eac001d60 con 0x7f5eb80a98f0 2024-03-25T19:46:31.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.342+0000 7f5eb57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:31.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.342+0000 7f5eb57fa640 1 -- 172.21.15.44:0/1305480252 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f5eac003790 con 0x7f5eb80a98f0 2024-03-25T19:46:31.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.342+0000 7f5eb57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:31.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.342+0000 7f5eb57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:31.342 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.342+0000 7f5eb57fa640 10 monclient: dump: 2024-03-25T19:46:31.342 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:31.342 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:31.343 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:31.343 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:31.343 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:31.343 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:31.343 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:31.343 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:31.343 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:31.343 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:31.343 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.342+0000 7f5eb6ffd640 4 set_mon_vals no callback set 2024-03-25T19:46:31.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.343+0000 7f5eb57fa640 1 -- 172.21.15.44:0/1305480252 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f5eac0039b0 con 0x7f5eb80a98f0 2024-03-25T19:46:31.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.344+0000 7f5eb7fff640 10 monclient: get_auth_request con 0x7f5e9c06df20 auth_method 0 2024-03-25T19:46:31.344 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.344+0000 7f5eb57fa640 10 client.4715.objecter ms_handle_connect 0x7f5e9c06df20 2024-03-25T19:46:31.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.351+0000 7f5eb57fa640 1 -- 172.21.15.44:0/1305480252 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1275227418 0 0) 0x7f5eac0534e0 con 0x7f5eb80a98f0 2024-03-25T19:46:31.350 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.351+0000 7f5eb57fa640 10 client.4715.objecter ms_dispatch 0x7f5eb8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:46:31.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.351+0000 7f5eb57fa640 3 client.4715.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:46:31.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.351+0000 7f5eb57fa640 3 client.4715.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:46:31.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.351+0000 7f5eb57fa640 20 client.4715.objecter dump_active .. 0 homeless 2024-03-25T19:46:31.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.351+0000 7f5eb57fa640 1 -- 172.21.15.44:0/1305480252 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5eac04eff0 con 0x7f5eb80a98f0 2024-03-25T19:46:31.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.351+0000 7f5eb57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:31.351 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.351+0000 7f5eb57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:31.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.605+0000 7f5ebf245640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:31.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.605+0000 7f5ebf245640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:31.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.605+0000 7f5ebf245640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:46:31.604 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.605+0000 7f5ebf245640 1 -- 172.21.15.44:0/1305480252 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5eb813b430 con 0x7f5eb80a98f0 2024-03-25T19:46:31.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.638+0000 7f5eb57fa640 1 -- 172.21.15.44:0/1305480252 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2782837909) 0x7f5eac050f20 con 0x7f5eb80a98f0 2024-03-25T19:46:31.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.638+0000 7f5eb57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:31.638 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.638+0000 7f5eb57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:31.638 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:46:31.638 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":11,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:45:30.548832+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:45:30.548832+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:31.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.641+0000 7f5ebf245640 10 librados: watch_flush enter 2024-03-25T19:46:31.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.641+0000 7f5ebf245640 10 librados: watch_flush exit 2024-03-25T19:46:31.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.641+0000 7f5ebf245640 20 client.4715.objecter shutdown clearing up homeless session... 2024-03-25T19:46:31.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.641+0000 7f5ebf245640 10 client.4715.objecter successfully canceled tick 2024-03-25T19:46:31.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.641+0000 7f5ebf245640 1 -- 172.21.15.44:0/1305480252 >> v1:172.21.15.111:6800/3883215793 conn(0x7f5e9c06df20 legacy=0x7f5e9c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:31.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.641+0000 7f5ebf245640 10 monclient: shutdown 2024-03-25T19:46:31.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.641+0000 7f5ebf245640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:31.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.641+0000 7f5ebf245640 1 -- 172.21.15.44:0/1305480252 >> v1:172.21.15.44:6790/0 conn(0x7f5eb80a98f0 legacy=0x7f5eb8133330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:31.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.641+0000 7f5ebf245640 1 -- 172.21.15.44:0/1305480252 shutdown_connections 2024-03-25T19:46:31.641 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.641+0000 7f5ebf245640 1 -- 172.21.15.44:0/1305480252 >> 172.21.15.44:0/1305480252 conn(0x7f5eb80b7ea0 msgr2=0x7f5eb8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:31.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.641+0000 7f5ebf245640 1 -- 172.21.15.44:0/1305480252 shutdown_connections 2024-03-25T19:46:31.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.641+0000 7f5ebf245640 1 -- 172.21.15.44:0/1305480252 wait complete. 2024-03-25T19:46:31.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:31.641+0000 7f5ebf245640 1 librados: shutdown 2024-03-25T19:46:31.652 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:38.459 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:38.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.563+0000 7fdc2e7c4640 10 monclient: get_monmap_and_config 2024-03-25T19:46:38.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.563+0000 7fdc2e7c4640 10 monclient: build_initial_monmap 2024-03-25T19:46:38.564 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.563+0000 7fdc2e7c4640 10 monclient: monmap: 2024-03-25T19:46:38.564 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:46:38.564 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:38.564 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:46:38.564967+0000 2024-03-25T19:46:38.564 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:46:38.564967+0000 2024-03-25T19:46:38.564 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:38.564 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:38.564 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:46:38.564 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:46:38.565 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:46:38.565 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:38.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.564+0000 7fdc2e7c4640 1 Processor -- start 2024-03-25T19:46:38.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.564+0000 7fdc2e7c4640 1 -- start start 2024-03-25T19:46:38.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.564+0000 7fdc2e7c4640 10 monclient: init 2024-03-25T19:46:38.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.565+0000 7fdc2e7c4640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:38.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.565+0000 7fdc2e7c4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:38.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.565+0000 7fdc2e7c4640 10 monclient(hunting): picked mon.noname-b con 0x7fdc280b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T19:46:38.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.565+0000 7fdc2e7c4640 10 monclient(hunting): picked mon.noname-c con 0x7fdc280b1730 addr v1:172.21.15.111:6789/0 2024-03-25T19:46:38.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.565+0000 7fdc2e7c4640 10 monclient(hunting): picked mon.noname-a con 0x7fdc2814be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:46:38.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.565+0000 7fdc2e7c4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc280af940 con 0x7fdc2814be90 2024-03-25T19:46:38.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.565+0000 7fdc2e7c4640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc280b0a70 con 0x7fdc280b1730 2024-03-25T19:46:38.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.565+0000 7fdc2e7c4640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc280b0d50 con 0x7fdc280b4d60 2024-03-25T19:46:38.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.565+0000 7fdc2e7c4640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:38.566 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.565+0000 7fdc2e7c4640 10 monclient(hunting): authenticate will time out at 6062.736266s 2024-03-25T19:46:38.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.574+0000 7fdc2cd61640 1 --1- >> v1:172.21.15.44:6789/0 conn(0x7fdc2814be90 0x7fdc2814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.44:6789/0 says I am v1:172.21.15.44:40286/0 (socket says 172.21.15.44:40286) 2024-03-25T19:46:38.574 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.574+0000 7fdc2cd61640 1 -- 172.21.15.44:0/2449710279 learned_addr learned my addr 172.21.15.44:0/2449710279 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:46:38.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.584+0000 7fdc26ffd640 1 -- 172.21.15.44:0/2449710279 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418966596 0 0) 0x7fdc280af940 con 0x7fdc2814be90 2024-03-25T19:46:38.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.584+0000 7fdc26ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:38.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.584+0000 7fdc26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:38.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.584+0000 7fdc26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:38.584 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.584+0000 7fdc26ffd640 10 monclient(hunting): my global_id is 4728 2024-03-25T19:46:38.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.584+0000 7fdc26ffd640 1 -- 172.21.15.44:0/2449710279 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc10003520 con 0x7fdc2814be90 2024-03-25T19:46:38.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.586+0000 7fdc26ffd640 1 -- 172.21.15.44:0/2449710279 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941308442 0 0) 0x7fdc280b0d50 con 0x7fdc280b4d60 2024-03-25T19:46:38.587 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.586+0000 7fdc26ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:38.587 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.586+0000 7fdc26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:38.587 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.586+0000 7fdc26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:38.587 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.586+0000 7fdc26ffd640 10 monclient(hunting): my global_id is 4718 2024-03-25T19:46:38.587 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.586+0000 7fdc26ffd640 1 -- 172.21.15.44:0/2449710279 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc280af940 con 0x7fdc280b4d60 2024-03-25T19:46:38.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.596+0000 7fdc26ffd640 1 -- 172.21.15.44:0/2449710279 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1244420854 0 0) 0x7fdc280b0a70 con 0x7fdc280b1730 2024-03-25T19:46:38.596 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.596+0000 7fdc26ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:38.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.596+0000 7fdc26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:38.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.596+0000 7fdc26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:38.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.596+0000 7fdc26ffd640 10 monclient(hunting): my global_id is 4726 2024-03-25T19:46:38.597 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.596+0000 7fdc26ffd640 1 -- 172.21.15.44:0/2449710279 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc280b0d50 con 0x7fdc280b1730 2024-03-25T19:46:38.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.607+0000 7fdc26ffd640 1 -- 172.21.15.44:0/2449710279 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1211615792 0 0) 0x7fdc280af940 con 0x7fdc280b4d60 2024-03-25T19:46:38.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.607+0000 7fdc26ffd640 1 -- 172.21.15.44:0/2449710279 >> v1:172.21.15.111:6789/0 conn(0x7fdc280b1730 legacy=0x7fdc280b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:38.607 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.607+0000 7fdc26ffd640 1 -- 172.21.15.44:0/2449710279 >> v1:172.21.15.44:6789/0 conn(0x7fdc2814be90 legacy=0x7fdc2814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:38.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.607+0000 7fdc26ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:38.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.607+0000 7fdc26ffd640 1 monclient: found mon.noname-b 2024-03-25T19:46:38.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.607+0000 7fdc26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:38.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.607+0000 7fdc26ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:46:38.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.607+0000 7fdc26ffd640 1 -- 172.21.15.44:0/2449710279 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc280b1030 con 0x7fdc280b4d60 2024-03-25T19:46:38.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.607+0000 7fdc26ffd640 10 monclient: _finish_auth 0 2024-03-25T19:46:38.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.607+0000 7fdc26ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:38.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.607+0000 7fdc26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:38.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.607+0000 7fdc26ffd640 1 -- 172.21.15.44:0/2449710279 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fdc18001670 con 0x7fdc280b4d60 2024-03-25T19:46:38.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.607+0000 7fdc26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:38.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.607+0000 7fdc26ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:38.608 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.607+0000 7fdc26ffd640 10 monclient: dump: 2024-03-25T19:46:38.608 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:38.608 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:38.609 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:38.609 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:38.609 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:38.609 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:38.609 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:38.609 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:38.609 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:38.609 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:38.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.607+0000 7fdc2e7c4640 5 monclient: authenticate success, global_id 4718 2024-03-25T19:46:38.609 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.607+0000 7fdc2e7c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:38.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.623+0000 7fdc26ffd640 1 -- 172.21.15.44:0/2449710279 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc180017d0 con 0x7fdc280b4d60 2024-03-25T19:46:38.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.623+0000 7fdc26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:38.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.623+0000 7fdc26ffd640 1 -- 172.21.15.44:0/2449710279 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fdc18001be0 con 0x7fdc280b4d60 2024-03-25T19:46:38.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.623+0000 7fdc2e7c4640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:38.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.623+0000 7fdc2e7c4640 4 set_mon_vals no callback set 2024-03-25T19:46:38.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.623+0000 7fdc26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:38.623 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.623+0000 7fdc26ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:38.624 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.623+0000 7fdc26ffd640 10 monclient: dump: 2024-03-25T19:46:38.624 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:38.624 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:38.624 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:38.624 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:38.624 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:38.624 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:38.624 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:38.624 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:38.624 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:38.624 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:38.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.624+0000 7fdc2e7c4640 10 monclient: shutdown 2024-03-25T19:46:38.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.624+0000 7fdc2e7c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:38.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.624+0000 7fdc2e7c4640 1 -- 172.21.15.44:0/2449710279 >> v1:172.21.15.44:6790/0 conn(0x7fdc280b4d60 legacy=0x7fdc2813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:38.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.624+0000 7fdc2e7c4640 1 -- 172.21.15.44:0/2449710279 shutdown_connections 2024-03-25T19:46:38.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.624+0000 7fdc2e7c4640 1 -- 172.21.15.44:0/2449710279 >> 172.21.15.44:0/2449710279 conn(0x7fdc280617a0 msgr2=0x7fdc28061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:38.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.624+0000 7fdc2e7c4640 1 -- 172.21.15.44:0/2449710279 shutdown_connections 2024-03-25T19:46:38.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.624+0000 7fdc2e7c4640 1 -- 172.21.15.44:0/2449710279 wait complete. 2024-03-25T19:46:38.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.625+0000 7fdc2e7c4640 10 monclient: build_initial_monmap 2024-03-25T19:46:38.626 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.625+0000 7fdc2e7c4640 10 monclient: monmap: 2024-03-25T19:46:38.626 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:46:38.626 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:38.627 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:46:38.627 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:46:38.627 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:38.627 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:46:38.627 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:46:38.627 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:46:38.627 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:46:38.627 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:38.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.625+0000 7fdc2e7c4640 1 librados: starting msgr at 2024-03-25T19:46:38.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.625+0000 7fdc2e7c4640 1 librados: starting objecter 2024-03-25T19:46:38.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.625+0000 7fdc2e7c4640 1 Processor -- start 2024-03-25T19:46:38.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.625+0000 7fdc2e7c4640 1 -- start start 2024-03-25T19:46:38.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.625+0000 7fdc2e7c4640 1 librados: setting wanted keys 2024-03-25T19:46:38.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.625+0000 7fdc2e7c4640 1 librados: calling monclient init 2024-03-25T19:46:38.627 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.625+0000 7fdc2e7c4640 10 monclient: init 2024-03-25T19:46:38.628 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.626+0000 7fdc2e7c4640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:38.628 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.626+0000 7fdc2e7c4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:38.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.626+0000 7fdc2e7c4640 10 monclient(hunting): picked mon.noname-a con 0x7fdc280b1730 addr v1:172.21.15.44:6789/0 2024-03-25T19:46:38.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.626+0000 7fdc2e7c4640 10 monclient(hunting): picked mon.noname-b con 0x7fdc280b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T19:46:38.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.626+0000 7fdc2e7c4640 10 monclient(hunting): picked mon.noname-c con 0x7fdc2814be90 addr v1:172.21.15.44:6790/0 2024-03-25T19:46:38.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.626+0000 7fdc2e7c4640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc28145370 con 0x7fdc280b1730 2024-03-25T19:46:38.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.626+0000 7fdc2e7c4640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc281455c0 con 0x7fdc280b4d60 2024-03-25T19:46:38.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.626+0000 7fdc2e7c4640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc281458a0 con 0x7fdc2814be90 2024-03-25T19:46:38.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.626+0000 7fdc2e7c4640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:38.629 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.626+0000 7fdc2e7c4640 10 monclient(hunting): authenticate will time out at 6062.797361s 2024-03-25T19:46:38.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.630+0000 7fdc277fe640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fdc280b4d60 0x7fdc2813d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:50008/0 (socket says 172.21.15.44:50008) 2024-03-25T19:46:38.630 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.630+0000 7fdc277fe640 1 -- 172.21.15.44:0/674739164 learned_addr learned my addr 172.21.15.44:0/674739164 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:46:38.631 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.631+0000 7fdc24ff9640 10 client.?.objecter ms_handle_connect 0x7fdc280b4d60 2024-03-25T19:46:38.631 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.631+0000 7fdc24ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:38.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.640+0000 7fdc24ff9640 1 -- 172.21.15.44:0/674739164 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816404559 0 0) 0x7fdc281455c0 con 0x7fdc280b4d60 2024-03-25T19:46:38.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.640+0000 7fdc24ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:38.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.640+0000 7fdc24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:38.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.640+0000 7fdc24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:38.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.640+0000 7fdc24ff9640 10 monclient(hunting): my global_id is 4729 2024-03-25T19:46:38.640 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.640+0000 7fdc24ff9640 1 -- 172.21.15.44:0/674739164 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc14004eb0 con 0x7fdc280b4d60 2024-03-25T19:46:38.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.642+0000 7fdc24ff9640 10 client.?.objecter ms_handle_connect 0x7fdc2814be90 2024-03-25T19:46:38.642 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.642+0000 7fdc24ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:38.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.647+0000 7fdc24ff9640 1 -- 172.21.15.44:0/674739164 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1217254261 0 0) 0x7fdc14004eb0 con 0x7fdc280b4d60 2024-03-25T19:46:38.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.647+0000 7fdc24ff9640 1 -- 172.21.15.44:0/674739164 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc281455c0 con 0x7fdc280b4d60 2024-03-25T19:46:38.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.647+0000 7fdc24ff9640 1 -- 172.21.15.44:0/674739164 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fdc1c001aa0 con 0x7fdc280b4d60 2024-03-25T19:46:38.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.647+0000 7fdc24ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:38.647 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.647+0000 7fdc24ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:38.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.647+0000 7fdc24ff9640 10 monclient(hunting): dump: 2024-03-25T19:46:38.648 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:38.648 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:38.648 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:38.648 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:38.648 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:38.648 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:38.648 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:38.648 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:38.648 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:38.648 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:38.648 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.647+0000 7fdc24ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:38.650 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.650+0000 7fdc24ff9640 10 client.?.objecter ms_handle_connect 0x7fdc280b1730 2024-03-25T19:46:38.650 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.650+0000 7fdc24ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:38.665 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.664+0000 7fdc24ff9640 1 -- 172.21.15.44:0/674739164 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1871150885 0 0) 0x7fdc281455c0 con 0x7fdc280b4d60 2024-03-25T19:46:38.665 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.665+0000 7fdc24ff9640 1 -- 172.21.15.44:0/674739164 >> v1:172.21.15.44:6790/0 conn(0x7fdc2814be90 legacy=0x7fdc2813d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:38.665 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.665+0000 7fdc24ff9640 1 -- 172.21.15.44:0/674739164 >> v1:172.21.15.44:6789/0 conn(0x7fdc280b1730 legacy=0x7fdc28140620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:38.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.665+0000 7fdc24ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:46:38.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.665+0000 7fdc24ff9640 1 monclient: found mon.b 2024-03-25T19:46:38.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.665+0000 7fdc24ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:38.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.665+0000 7fdc24ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:46:38.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.665+0000 7fdc24ff9640 1 -- 172.21.15.44:0/674739164 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2813f030 con 0x7fdc280b4d60 2024-03-25T19:46:38.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.665+0000 7fdc24ff9640 10 monclient: _finish_auth 0 2024-03-25T19:46:38.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.665+0000 7fdc24ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:46:38.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.665+0000 7fdc24ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:38.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.665+0000 7fdc2e7c4640 5 monclient: authenticate success, global_id 4729 2024-03-25T19:46:38.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.665+0000 7fdc2e7c4640 10 monclient: _renew_subs 2024-03-25T19:46:38.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.665+0000 7fdc2e7c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:46:38.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.665+0000 7fdc2e7c4640 1 -- 172.21.15.44:0/674739164 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc28143900 con 0x7fdc280b4d60 2024-03-25T19:46:38.666 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.665+0000 7fdc2e7c4640 10 client.4729.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:38.667 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.665+0000 7fdc2e7c4640 10 monclient: _renew_subs 2024-03-25T19:46:38.667 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.665+0000 7fdc2e7c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:46:38.667 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.665+0000 7fdc2e7c4640 1 -- 172.21.15.44:0/674739164 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2813e0f0 con 0x7fdc280b4d60 2024-03-25T19:46:38.667 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.665+0000 7fdc2e7c4640 1 librados: init done 2024-03-25T19:46:38.667 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.666+0000 7fdc2e7c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:38.667 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.666+0000 7fdc2e7c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:38.668 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.666+0000 7fdc2e7c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:46:38.668 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.666+0000 7fdc2e7c4640 1 -- 172.21.15.44:0/674739164 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc2813e340 con 0x7fdc280b4d60 2024-03-25T19:46:38.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.944+0000 7fdc24ff9640 1 -- 172.21.15.44:0/674739164 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc1c0038c0 con 0x7fdc280b4d60 2024-03-25T19:46:38.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.944+0000 7fdc24ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:38.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.944+0000 7fdc24ff9640 1 -- 172.21.15.44:0/674739164 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7fdc1c001c00 con 0x7fdc280b4d60 2024-03-25T19:46:38.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.944+0000 7fdc24ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:38.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.944+0000 7fdc24ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:38.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.944+0000 7fdc24ff9640 10 monclient: dump: 2024-03-25T19:46:38.945 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:38.945 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:38.945 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:38.945 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:38.945 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:38.946 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:38.946 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:38.946 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:38.946 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:38.946 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:38.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.944+0000 7fdc267fc640 4 set_mon_vals no callback set 2024-03-25T19:46:38.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.945+0000 7fdc24ff9640 1 -- 172.21.15.44:0/674739164 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7fdc1c003d90 con 0x7fdc280b4d60 2024-03-25T19:46:38.947 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.946+0000 7fdc27fff640 10 monclient: get_auth_request con 0x7fdc1406dd00 auth_method 0 2024-03-25T19:46:38.947 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.946+0000 7fdc24ff9640 10 client.4729.objecter ms_handle_connect 0x7fdc1406dd00 2024-03-25T19:46:38.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.953+0000 7fdc24ff9640 1 -- 172.21.15.44:0/674739164 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1275227418 0 0) 0x7fdc1c0548b0 con 0x7fdc280b4d60 2024-03-25T19:46:38.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.953+0000 7fdc24ff9640 10 client.4729.objecter ms_dispatch 0x7fdc28000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:46:38.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.953+0000 7fdc24ff9640 3 client.4729.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:46:38.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.953+0000 7fdc24ff9640 3 client.4729.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:46:38.953 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.953+0000 7fdc24ff9640 20 client.4729.objecter dump_active .. 0 homeless 2024-03-25T19:46:38.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.954+0000 7fdc24ff9640 1 -- 172.21.15.44:0/674739164 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdc1c0503c0 con 0x7fdc280b4d60 2024-03-25T19:46:38.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.954+0000 7fdc24ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:38.954 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:38.954+0000 7fdc24ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:39.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:39.213+0000 7fdc2e7c4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:39.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:39.213+0000 7fdc2e7c4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:39.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:39.213+0000 7fdc2e7c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:46:39.214 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:39.213+0000 7fdc2e7c4640 1 -- 172.21.15.44:0/674739164 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc280627b0 con 0x7fdc280b4d60 2024-03-25T19:46:39.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:39.240+0000 7fdc24ff9640 1 -- 172.21.15.44:0/674739164 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2782837909) 0x7fdc1c0522f0 con 0x7fdc280b4d60 2024-03-25T19:46:39.240 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:39.240+0000 7fdc24ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:39.241 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:39.240+0000 7fdc24ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:39.241 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:46:39.241 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":11,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:45:30.548832+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:45:30.548832+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:39.243 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:39.243+0000 7fdc2e7c4640 10 librados: watch_flush enter 2024-03-25T19:46:39.243 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:39.243+0000 7fdc2e7c4640 10 librados: watch_flush exit 2024-03-25T19:46:39.243 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:39.243+0000 7fdc2e7c4640 20 client.4729.objecter shutdown clearing up homeless session... 2024-03-25T19:46:39.243 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:39.243+0000 7fdc2e7c4640 10 client.4729.objecter successfully canceled tick 2024-03-25T19:46:39.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:39.243+0000 7fdc2e7c4640 1 -- 172.21.15.44:0/674739164 >> v1:172.21.15.111:6800/3883215793 conn(0x7fdc1406dd00 legacy=0x7fdc140701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:39.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:39.243+0000 7fdc2e7c4640 10 monclient: shutdown 2024-03-25T19:46:39.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:39.243+0000 7fdc2e7c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:39.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:39.243+0000 7fdc2e7c4640 1 -- 172.21.15.44:0/674739164 >> v1:172.21.15.111:6789/0 conn(0x7fdc280b4d60 legacy=0x7fdc2813d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:39.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:39.243+0000 7fdc2e7c4640 1 -- 172.21.15.44:0/674739164 shutdown_connections 2024-03-25T19:46:39.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:39.243+0000 7fdc2e7c4640 1 -- 172.21.15.44:0/674739164 >> 172.21.15.44:0/674739164 conn(0x7fdc280617a0 msgr2=0x7fdc280b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:39.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:39.243+0000 7fdc2e7c4640 1 -- 172.21.15.44:0/674739164 shutdown_connections 2024-03-25T19:46:39.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:39.243+0000 7fdc2e7c4640 1 -- 172.21.15.44:0/674739164 wait complete. 2024-03-25T19:46:39.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:39.243+0000 7fdc2e7c4640 1 librados: shutdown 2024-03-25T19:46:39.256 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:39.611 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:11:30.513 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:39.619 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:11:30.521 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:39.629 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:11:30.530 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:39.632 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:11:30.534 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:39.645 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:11:30.546 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:39.650 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:11:30.530 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:39.654 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:11:30.534 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:39.698 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:11:30.600 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:39.716 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:11:30.596 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:46.063 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:46.164 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:11:37.065 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:46.165 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.165+0000 7f6f5213d640 10 monclient: get_monmap_and_config 2024-03-25T19:46:46.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.165+0000 7f6f5213d640 10 monclient: build_initial_monmap 2024-03-25T19:46:46.166 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.165+0000 7f6f5213d640 10 monclient: monmap: 2024-03-25T19:46:46.166 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:46:46.166 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:46.166 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:46:46.167005+0000 2024-03-25T19:46:46.166 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:46:46.167005+0000 2024-03-25T19:46:46.166 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:46.166 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:46.166 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:46:46.166 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:46:46.166 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:46:46.166 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:46.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.166+0000 7f6f5213d640 1 Processor -- start 2024-03-25T19:46:46.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.166+0000 7f6f5213d640 1 -- start start 2024-03-25T19:46:46.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.166+0000 7f6f5213d640 10 monclient: init 2024-03-25T19:46:46.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.167+0000 7f6f5213d640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:46.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.167+0000 7f6f5213d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:46.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.167+0000 7f6f5213d640 10 monclient(hunting): picked mon.noname-b con 0x7f6f4c0b4d60 addr v1:172.21.15.44:6790/0 2024-03-25T19:46:46.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.167+0000 7f6f5213d640 10 monclient(hunting): picked mon.noname-c con 0x7f6f4c0b1730 addr v1:172.21.15.111:6789/0 2024-03-25T19:46:46.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.167+0000 7f6f5213d640 10 monclient(hunting): picked mon.noname-a con 0x7f6f4c14be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:46:46.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.167+0000 7f6f5213d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f4c0af940 con 0x7f6f4c14be90 2024-03-25T19:46:46.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.167+0000 7f6f5213d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f4c0b0a70 con 0x7f6f4c0b1730 2024-03-25T19:46:46.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.167+0000 7f6f5213d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f4c0b0d50 con 0x7f6f4c0b4d60 2024-03-25T19:46:46.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.167+0000 7f6f5213d640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:46.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.167+0000 7f6f5213d640 10 monclient(hunting): authenticate will time out at 6070.338124s 2024-03-25T19:46:46.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.167+0000 7f6f4affd640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f6f4c0b1730 0x7f6f4c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:54502/0 (socket says 172.21.15.44:54502) 2024-03-25T19:46:46.168 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.167+0000 7f6f4affd640 1 -- 172.21.15.44:0/2593660007 learned_addr learned my addr 172.21.15.44:0/2593660007 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:46:46.173 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:11:37.075 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:46.206 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:11:37.108 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:46.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.212+0000 7f6f4a7fc640 1 -- 172.21.15.44:0/2593660007 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983185910 0 0) 0x7f6f4c0af940 con 0x7f6f4c14be90 2024-03-25T19:46:46.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.212+0000 7f6f4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:46.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.212+0000 7f6f4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:46.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.212+0000 7f6f4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:46.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.212+0000 7f6f4a7fc640 10 monclient(hunting): my global_id is 4734 2024-03-25T19:46:46.213 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.212+0000 7f6f4a7fc640 1 -- 172.21.15.44:0/2593660007 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f34003520 con 0x7f6f4c14be90 2024-03-25T19:46:46.230 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:11:37.132 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:46.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.231+0000 7f6f4a7fc640 1 -- 172.21.15.44:0/2593660007 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605708055 0 0) 0x7f6f4c0b0d50 con 0x7f6f4c0b4d60 2024-03-25T19:46:46.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.231+0000 7f6f4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:46.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.231+0000 7f6f4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:46.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.231+0000 7f6f4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:46.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.231+0000 7f6f4a7fc640 10 monclient(hunting): my global_id is 4724 2024-03-25T19:46:46.231 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.231+0000 7f6f4a7fc640 1 -- 172.21.15.44:0/2593660007 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f4c0af940 con 0x7f6f4c0b4d60 2024-03-25T19:46:46.235 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:11:37.136 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:46.240 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:11:37.119 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:46.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.252+0000 7f6f4a7fc640 1 -- 172.21.15.44:0/2593660007 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253885229 0 0) 0x7f6f4c0b0a70 con 0x7f6f4c0b1730 2024-03-25T19:46:46.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.252+0000 7f6f4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:46.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.252+0000 7f6f4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:46.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.252+0000 7f6f4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:46.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.252+0000 7f6f4a7fc640 10 monclient(hunting): my global_id is 4732 2024-03-25T19:46:46.252 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.252+0000 7f6f4a7fc640 1 -- 172.21.15.44:0/2593660007 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f4c0b0d50 con 0x7f6f4c0b1730 2024-03-25T19:46:46.256 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:11:37.136 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:46.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.257+0000 7f6f4a7fc640 1 -- 172.21.15.44:0/2593660007 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 260699593 0 0) 0x7f6f4c0af940 con 0x7f6f4c0b4d60 2024-03-25T19:46:46.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.257+0000 7f6f4a7fc640 1 -- 172.21.15.44:0/2593660007 >> v1:172.21.15.111:6789/0 conn(0x7f6f4c0b1730 legacy=0x7f6f4c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:46.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.257+0000 7f6f4a7fc640 1 -- 172.21.15.44:0/2593660007 >> v1:172.21.15.44:6789/0 conn(0x7f6f4c14be90 legacy=0x7f6f4c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:46.257 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.257+0000 7f6f4a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:46:46.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.257+0000 7f6f4a7fc640 1 monclient: found mon.noname-b 2024-03-25T19:46:46.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.257+0000 7f6f4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:46.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.257+0000 7f6f4a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.44:6790/0 2024-03-25T19:46:46.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.257+0000 7f6f4a7fc640 1 -- 172.21.15.44:0/2593660007 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4c0b1030 con 0x7f6f4c0b4d60 2024-03-25T19:46:46.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.257+0000 7f6f4a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:46:46.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.257+0000 7f6f4a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:46:46.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.257+0000 7f6f4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:46.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.257+0000 7f6f4a7fc640 1 -- 172.21.15.44:0/2593660007 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6f3c001670 con 0x7f6f4c0b4d60 2024-03-25T19:46:46.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.257+0000 7f6f4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:46.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.257+0000 7f6f4a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:46.258 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.257+0000 7f6f4a7fc640 10 monclient: dump: 2024-03-25T19:46:46.258 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:46.258 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:46.259 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:46.259 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:46.259 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:46.259 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:46.259 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:46.259 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:46.259 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:46.259 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:46.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.257+0000 7f6f5213d640 5 monclient: authenticate success, global_id 4724 2024-03-25T19:46:46.259 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.257+0000 7f6f5213d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:46.270 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:11:37.171 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:46.279 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:11:37.158 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:46.282 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.282+0000 7f6f4a7fc640 1 -- 172.21.15.44:0/2593660007 <== mon.2 v1:172.21.15.44:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f3c0017d0 con 0x7f6f4c0b4d60 2024-03-25T19:46:46.282 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.282+0000 7f6f4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:46.282 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.282+0000 7f6f4a7fc640 1 -- 172.21.15.44:0/2593660007 <== mon.2 v1:172.21.15.44:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6f3c001be0 con 0x7f6f4c0b4d60 2024-03-25T19:46:46.282 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.282+0000 7f6f5213d640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:46.282 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.282+0000 7f6f5213d640 4 set_mon_vals no callback set 2024-03-25T19:46:46.282 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.282+0000 7f6f4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:46.282 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.282+0000 7f6f4a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:46.282 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.282+0000 7f6f4a7fc640 10 monclient: dump: 2024-03-25T19:46:46.282 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:46.283 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:46.283 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:46.283 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:46.283 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:46.283 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:46.283 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:46.283 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:46.283 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:46.283 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:46.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.282+0000 7f6f5213d640 10 monclient: shutdown 2024-03-25T19:46:46.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.282+0000 7f6f5213d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:46.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.282+0000 7f6f5213d640 1 -- 172.21.15.44:0/2593660007 >> v1:172.21.15.44:6790/0 conn(0x7f6f4c0b4d60 legacy=0x7f6f4c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:46.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.282+0000 7f6f5213d640 1 -- 172.21.15.44:0/2593660007 shutdown_connections 2024-03-25T19:46:46.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.282+0000 7f6f5213d640 1 -- 172.21.15.44:0/2593660007 >> 172.21.15.44:0/2593660007 conn(0x7f6f4c0617a0 msgr2=0x7f6f4c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:46.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.283+0000 7f6f5213d640 1 -- 172.21.15.44:0/2593660007 shutdown_connections 2024-03-25T19:46:46.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.283+0000 7f6f5213d640 1 -- 172.21.15.44:0/2593660007 wait complete. 2024-03-25T19:46:46.286 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.283+0000 7f6f5213d640 10 monclient: build_initial_monmap 2024-03-25T19:46:46.286 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.283+0000 7f6f5213d640 10 monclient: monmap: 2024-03-25T19:46:46.286 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:46:46.286 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:46.286 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:46:46.286 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:46:46.286 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:46.286 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:46:46.286 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:46:46.286 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:46:46.286 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:46:46.286 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:46.286 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.284+0000 7f6f5213d640 1 librados: starting msgr at 2024-03-25T19:46:46.286 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.284+0000 7f6f5213d640 1 librados: starting objecter 2024-03-25T19:46:46.286 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.284+0000 7f6f5213d640 1 Processor -- start 2024-03-25T19:46:46.287 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.284+0000 7f6f5213d640 1 -- start start 2024-03-25T19:46:46.287 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.284+0000 7f6f5213d640 1 librados: setting wanted keys 2024-03-25T19:46:46.287 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.284+0000 7f6f5213d640 1 librados: calling monclient init 2024-03-25T19:46:46.287 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.284+0000 7f6f5213d640 10 monclient: init 2024-03-25T19:46:46.287 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.284+0000 7f6f5213d640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:46.287 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.285+0000 7f6f5213d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:46.287 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.285+0000 7f6f5213d640 10 monclient(hunting): picked mon.noname-c con 0x7f6f4c0b1730 addr v1:172.21.15.44:6790/0 2024-03-25T19:46:46.287 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.285+0000 7f6f5213d640 10 monclient(hunting): picked mon.noname-b con 0x7f6f4c0b4d60 addr v1:172.21.15.111:6789/0 2024-03-25T19:46:46.287 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.285+0000 7f6f5213d640 10 monclient(hunting): picked mon.noname-a con 0x7f6f4c14be90 addr v1:172.21.15.44:6789/0 2024-03-25T19:46:46.287 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.285+0000 7f6f5213d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f4c145370 con 0x7f6f4c14be90 2024-03-25T19:46:46.287 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.285+0000 7f6f5213d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f4c1455c0 con 0x7f6f4c0b4d60 2024-03-25T19:46:46.287 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.285+0000 7f6f5213d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f4c1458a0 con 0x7f6f4c0b1730 2024-03-25T19:46:46.287 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.285+0000 7f6f5213d640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:46.287 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.285+0000 7f6f5213d640 10 monclient(hunting): authenticate will time out at 6070.456083s 2024-03-25T19:46:46.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.298+0000 7f6f4b7fe640 1 --1- >> v1:172.21.15.44:6790/0 conn(0x7f6f4c0b1730 0x7f6f4c140620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.44:6790/0 says I am v1:172.21.15.44:49850/0 (socket says 172.21.15.44:49850) 2024-03-25T19:46:46.298 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.298+0000 7f6f4b7fe640 1 -- 172.21.15.44:0/252363196 learned_addr learned my addr 172.21.15.44:0/252363196 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:46:46.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.305+0000 7f6f33fff640 10 client.?.objecter ms_handle_connect 0x7f6f4c0b1730 2024-03-25T19:46:46.305 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.305+0000 7f6f33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:46.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.321+0000 7f6f33fff640 1 -- 172.21.15.44:0/252363196 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 67409809 0 0) 0x7f6f4c1458a0 con 0x7f6f4c0b1730 2024-03-25T19:46:46.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.321+0000 7f6f33fff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:46.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.321+0000 7f6f33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:46.321 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.321+0000 7f6f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:46.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.321+0000 7f6f33fff640 10 monclient(hunting): my global_id is 4727 2024-03-25T19:46:46.322 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.321+0000 7f6f33fff640 1 -- 172.21.15.44:0/252363196 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f38004eb0 con 0x7f6f4c0b1730 2024-03-25T19:46:46.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.344+0000 7f6f33fff640 10 client.?.objecter ms_handle_connect 0x7f6f4c14be90 2024-03-25T19:46:46.345 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.345+0000 7f6f33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:46.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.355+0000 7f6f33fff640 1 -- 172.21.15.44:0/252363196 <== mon.2 v1:172.21.15.44:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2457711338 0 0) 0x7f6f38004eb0 con 0x7f6f4c0b1730 2024-03-25T19:46:46.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.355+0000 7f6f33fff640 1 -- 172.21.15.44:0/252363196 --> v1:172.21.15.44:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f4c1458a0 con 0x7f6f4c0b1730 2024-03-25T19:46:46.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.355+0000 7f6f33fff640 1 -- 172.21.15.44:0/252363196 <== mon.2 v1:172.21.15.44:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6f3c001670 con 0x7f6f4c0b1730 2024-03-25T19:46:46.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.355+0000 7f6f33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:46.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.355+0000 7f6f33fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:46.356 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.355+0000 7f6f33fff640 10 monclient(hunting): dump: 2024-03-25T19:46:46.356 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:46.356 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:46.356 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:46.356 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:46.356 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:46.356 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:46.356 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:46.357 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:46.357 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:46.357 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:46.357 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.355+0000 7f6f33fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:46.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.358+0000 7f6f33fff640 1 -- 172.21.15.44:0/252363196 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025735280 0 0) 0x7f6f4c145370 con 0x7f6f4c14be90 2024-03-25T19:46:46.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.358+0000 7f6f33fff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:46.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.358+0000 7f6f33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:46.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.358+0000 7f6f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:46.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.358+0000 7f6f33fff640 10 monclient(hunting): my global_id is 4737 2024-03-25T19:46:46.358 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.358+0000 7f6f33fff640 1 -- 172.21.15.44:0/252363196 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f38004eb0 con 0x7f6f4c14be90 2024-03-25T19:46:46.365 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.365+0000 7f6f33fff640 10 client.?.objecter ms_handle_connect 0x7f6f4c0b4d60 2024-03-25T19:46:46.366 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.365+0000 7f6f33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:46.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.377+0000 7f6f33fff640 1 -- 172.21.15.44:0/252363196 <== mon.2 v1:172.21.15.44:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 716170882 0 0) 0x7f6f4c1458a0 con 0x7f6f4c0b1730 2024-03-25T19:46:46.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.377+0000 7f6f33fff640 1 -- 172.21.15.44:0/252363196 >> v1:172.21.15.111:6789/0 conn(0x7f6f4c0b4d60 legacy=0x7f6f4c13d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:46.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.377+0000 7f6f33fff640 1 -- 172.21.15.44:0/252363196 >> v1:172.21.15.44:6789/0 conn(0x7f6f4c14be90 legacy=0x7f6f4c13d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:46.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.377+0000 7f6f33fff640 10 monclient: _finish_hunting 0 2024-03-25T19:46:46.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.377+0000 7f6f33fff640 1 monclient: found mon.c 2024-03-25T19:46:46.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.377+0000 7f6f33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:46.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.377+0000 7f6f33fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:46:46.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.377+0000 7f6f33fff640 1 -- 172.21.15.44:0/252363196 --> v1:172.21.15.44:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4c13f030 con 0x7f6f4c0b1730 2024-03-25T19:46:46.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.377+0000 7f6f33fff640 10 monclient: _finish_auth 0 2024-03-25T19:46:46.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.377+0000 7f6f33fff640 10 monclient: _check_auth_tickets 2024-03-25T19:46:46.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.377+0000 7f6f33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:46.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.377+0000 7f6f5213d640 5 monclient: authenticate success, global_id 4727 2024-03-25T19:46:46.378 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.377+0000 7f6f5213d640 10 monclient: _renew_subs 2024-03-25T19:46:46.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.377+0000 7f6f5213d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:46:46.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.377+0000 7f6f5213d640 1 -- 172.21.15.44:0/252363196 --> v1:172.21.15.44:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f4c143900 con 0x7f6f4c0b1730 2024-03-25T19:46:46.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.377+0000 7f6f5213d640 10 client.4727.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:46.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.377+0000 7f6f5213d640 10 monclient: _renew_subs 2024-03-25T19:46:46.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.377+0000 7f6f5213d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:46:46.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.377+0000 7f6f5213d640 1 -- 172.21.15.44:0/252363196 --> v1:172.21.15.44:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f4c13e0f0 con 0x7f6f4c0b1730 2024-03-25T19:46:46.379 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.377+0000 7f6f5213d640 1 librados: init done 2024-03-25T19:46:46.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.378+0000 7f6f5213d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:46.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.378+0000 7f6f5213d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:46.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.378+0000 7f6f5213d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:46:46.380 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.378+0000 7f6f5213d640 1 -- 172.21.15.44:0/252363196 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f4c13e340 con 0x7f6f4c0b1730 2024-03-25T19:46:46.712 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.712+0000 7f6f33fff640 1 -- 172.21.15.44:0/252363196 <== mon.2 v1:172.21.15.44:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f3c0038a0 con 0x7f6f4c0b1730 2024-03-25T19:46:46.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.712+0000 7f6f33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:46.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.712+0000 7f6f33fff640 1 -- 172.21.15.44:0/252363196 <== mon.2 v1:172.21.15.44:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f6f3c003c70 con 0x7f6f4c0b1730 2024-03-25T19:46:46.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.712+0000 7f6f33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:46.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.712+0000 7f6f33fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:46.713 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.712+0000 7f6f33fff640 10 monclient: dump: 2024-03-25T19:46:46.713 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:46.713 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:46.713 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:46.713 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:46.713 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:46.713 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:46.713 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:46.714 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:46.714 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:46.714 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:46.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.712+0000 7f6f49ffb640 4 set_mon_vals no callback set 2024-03-25T19:46:46.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.713+0000 7f6f33fff640 1 -- 172.21.15.44:0/252363196 <== mon.2 v1:172.21.15.44:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f6f3c003e90 con 0x7f6f4c0b1730 2024-03-25T19:46:46.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.713+0000 7f6f4affd640 10 monclient: get_auth_request con 0x7f6f3806dd90 auth_method 0 2024-03-25T19:46:46.714 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.714+0000 7f6f33fff640 10 client.4727.objecter ms_handle_connect 0x7f6f3806dd90 2024-03-25T19:46:46.721 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.721+0000 7f6f33fff640 1 -- 172.21.15.44:0/252363196 <== mon.2 v1:172.21.15.44:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1275227418 0 0) 0x7f6f3c054950 con 0x7f6f4c0b1730 2024-03-25T19:46:46.721 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.721+0000 7f6f33fff640 10 client.4727.objecter ms_dispatch 0x7f6f4c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:46:46.721 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.721+0000 7f6f33fff640 3 client.4727.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:46:46.721 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.721+0000 7f6f33fff640 3 client.4727.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:46:46.722 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.721+0000 7f6f33fff640 20 client.4727.objecter dump_active .. 0 homeless 2024-03-25T19:46:46.722 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.721+0000 7f6f33fff640 1 -- 172.21.15.44:0/252363196 <== mon.2 v1:172.21.15.44:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6f3c050460 con 0x7f6f4c0b1730 2024-03-25T19:46:46.722 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.721+0000 7f6f33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:46.722 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.721+0000 7f6f33fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:46.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.973+0000 7f6f5213d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:46.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.973+0000 7f6f5213d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:46.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.973+0000 7f6f5213d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.44:6790/0 2024-03-25T19:46:46.974 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:46.973+0000 7f6f5213d640 1 -- 172.21.15.44:0/252363196 --> v1:172.21.15.44:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f4c0627b0 con 0x7f6f4c0b1730 2024-03-25T19:46:47.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:47.008+0000 7f6f33fff640 1 -- 172.21.15.44:0/252363196 <== mon.2 v1:172.21.15.44:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2782837909) 0x7f6f3c052390 con 0x7f6f4c0b1730 2024-03-25T19:46:47.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:47.008+0000 7f6f33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:47.009 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:47.008+0000 7f6f33fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:47.010 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:46:47.010 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":11,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:45:30.548832+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:45:30.548832+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:47.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:47.010+0000 7f6f5213d640 10 librados: watch_flush enter 2024-03-25T19:46:47.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:47.010+0000 7f6f5213d640 10 librados: watch_flush exit 2024-03-25T19:46:47.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:47.010+0000 7f6f5213d640 20 client.4727.objecter shutdown clearing up homeless session... 2024-03-25T19:46:47.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:47.010+0000 7f6f5213d640 10 client.4727.objecter successfully canceled tick 2024-03-25T19:46:47.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:47.010+0000 7f6f5213d640 1 -- 172.21.15.44:0/252363196 >> v1:172.21.15.111:6800/3883215793 conn(0x7f6f3806dd90 legacy=0x7f6f38070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:47.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:47.010+0000 7f6f5213d640 10 monclient: shutdown 2024-03-25T19:46:47.011 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:47.011+0000 7f6f5213d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:47.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:47.011+0000 7f6f5213d640 1 -- 172.21.15.44:0/252363196 >> v1:172.21.15.44:6790/0 conn(0x7f6f4c0b1730 legacy=0x7f6f4c140620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:47.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:47.011+0000 7f6f5213d640 1 -- 172.21.15.44:0/252363196 shutdown_connections 2024-03-25T19:46:47.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:47.011+0000 7f6f5213d640 1 -- 172.21.15.44:0/252363196 >> 172.21.15.44:0/252363196 conn(0x7f6f4c0617a0 msgr2=0x7f6f4c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:47.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:47.011+0000 7f6f5213d640 1 -- 172.21.15.44:0/252363196 shutdown_connections 2024-03-25T19:46:47.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:47.011+0000 7f6f5213d640 1 -- 172.21.15.44:0/252363196 wait complete. 2024-03-25T19:46:47.012 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:47.011+0000 7f6f5213d640 1 librados: shutdown 2024-03-25T19:46:47.023 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:53.830 DEBUG:teuthology.orchestra.run.smithi044:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:53.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.943+0000 7f58e159d640 10 monclient: get_monmap_and_config 2024-03-25T19:46:53.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.943+0000 7f58e159d640 10 monclient: build_initial_monmap 2024-03-25T19:46:53.943 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.943+0000 7f58e159d640 10 monclient: monmap: 2024-03-25T19:46:53.943 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:46:53.944 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:53.944 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:46:53.944439+0000 2024-03-25T19:46:53.944 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:46:53.944439+0000 2024-03-25T19:46:53.944 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:53.944 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:53.944 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:46:53.944 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-c 2024-03-25T19:46:53.944 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-b 2024-03-25T19:46:53.944 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:53.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.943+0000 7f58e159d640 1 Processor -- start 2024-03-25T19:46:53.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.944+0000 7f58e159d640 1 -- start start 2024-03-25T19:46:53.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.944+0000 7f58e159d640 10 monclient: init 2024-03-25T19:46:53.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.944+0000 7f58e159d640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:53.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.944+0000 7f58e159d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:53.945 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.944+0000 7f58e159d640 10 monclient(hunting): picked mon.noname-b con 0x7f58dc0a98f0 addr v1:172.21.15.44:6790/0 2024-03-25T19:46:53.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.944+0000 7f58e159d640 10 monclient(hunting): picked mon.noname-c con 0x7f58dc13f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:46:53.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.944+0000 7f58e159d640 10 monclient(hunting): picked mon.noname-a con 0x7f58dc1430c0 addr v1:172.21.15.44:6789/0 2024-03-25T19:46:53.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.944+0000 7f58e159d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58dc142520 con 0x7f58dc1430c0 2024-03-25T19:46:53.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.944+0000 7f58e159d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58dc0b28a0 con 0x7f58dc13f710 2024-03-25T19:46:53.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.944+0000 7f58e159d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58dc0b2a30 con 0x7f58dc0a98f0 2024-03-25T19:46:53.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.944+0000 7f58e159d640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:53.946 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.944+0000 7f58e159d640 10 monclient(hunting): authenticate will time out at 6078.115696s 2024-03-25T19:46:53.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.952+0000 7f58da7fc640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f58dc13f710 0x7f58dc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:49876/0 (socket says 172.21.15.44:49876) 2024-03-25T19:46:53.952 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.952+0000 7f58da7fc640 1 -- 172.21.15.44:0/1644960571 learned_addr learned my addr 172.21.15.44:0/1644960571 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:46:53.955 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.955+0000 7f58d9ffb640 1 -- 172.21.15.44:0/1644960571 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676723432 0 0) 0x7f58dc0b28a0 con 0x7f58dc13f710 2024-03-25T19:46:53.955 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.955+0000 7f58d9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:53.955 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.955+0000 7f58d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:53.955 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.955+0000 7f58d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:53.955 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.955+0000 7f58d9ffb640 10 monclient(hunting): my global_id is 4738 2024-03-25T19:46:53.956 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.955+0000 7f58d9ffb640 1 -- 172.21.15.44:0/1644960571 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58c4003520 con 0x7f58dc13f710 2024-03-25T19:46:53.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.963+0000 7f58d9ffb640 1 -- 172.21.15.44:0/1644960571 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1354518499 0 0) 0x7f58c4003520 con 0x7f58dc13f710 2024-03-25T19:46:53.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.964+0000 7f58d9ffb640 1 -- 172.21.15.44:0/1644960571 >> v1:172.21.15.44:6790/0 conn(0x7f58dc0a98f0 legacy=0x7f58dc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:53.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.964+0000 7f58d9ffb640 1 -- 172.21.15.44:0/1644960571 >> v1:172.21.15.44:6789/0 conn(0x7f58dc1430c0 legacy=0x7f58dc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:46:53.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.964+0000 7f58d9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:46:53.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.964+0000 7f58d9ffb640 1 monclient: found mon.noname-c 2024-03-25T19:46:53.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.964+0000 7f58d9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:53.964 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.964+0000 7f58d9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.111:6789/0 2024-03-25T19:46:53.965 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.964+0000 7f58d9ffb640 1 -- 172.21.15.44:0/1644960571 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58dc0b44c0 con 0x7f58dc13f710 2024-03-25T19:46:53.965 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.964+0000 7f58d9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:46:53.965 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.964+0000 7f58d9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:46:53.965 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.964+0000 7f58d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:53.965 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.964+0000 7f58d9ffb640 1 -- 172.21.15.44:0/1644960571 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f58d0001690 con 0x7f58dc13f710 2024-03-25T19:46:53.965 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.964+0000 7f58d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:53.965 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.964+0000 7f58d9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:53.965 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.964+0000 7f58d9ffb640 10 monclient: dump: 2024-03-25T19:46:53.965 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:53.965 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:53.965 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:53.965 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:53.965 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:53.965 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:53.965 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:53.966 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:53.966 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:53.966 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:53.966 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.964+0000 7f58e159d640 5 monclient: authenticate success, global_id 4738 2024-03-25T19:46:53.966 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.964+0000 7f58e159d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:53.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.967+0000 7f58d9ffb640 1 -- 172.21.15.44:0/1644960571 <== mon.1 v1:172.21.15.111:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58d0001920 con 0x7f58dc13f710 2024-03-25T19:46:53.967 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.967+0000 7f58d9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:53.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.967+0000 7f58e159d640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:53.968 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.967+0000 7f58e159d640 4 set_mon_vals no callback set 2024-03-25T19:46:53.969 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.968+0000 7f58e159d640 10 monclient: shutdown 2024-03-25T19:46:53.969 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.968+0000 7f58e159d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:53.969 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.968+0000 7f58e159d640 1 -- 172.21.15.44:0/1644960571 >> v1:172.21.15.111:6789/0 conn(0x7f58dc13f710 legacy=0x7f58dc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:53.969 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.968+0000 7f58e159d640 1 -- 172.21.15.44:0/1644960571 shutdown_connections 2024-03-25T19:46:53.969 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.968+0000 7f58e159d640 1 -- 172.21.15.44:0/1644960571 >> 172.21.15.44:0/1644960571 conn(0x7f58dc0b7ea0 msgr2=0x7f58dc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:53.969 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.968+0000 7f58e159d640 1 -- 172.21.15.44:0/1644960571 shutdown_connections 2024-03-25T19:46:53.969 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.968+0000 7f58e159d640 1 -- 172.21.15.44:0/1644960571 wait complete. 2024-03-25T19:46:53.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.969+0000 7f58e159d640 10 monclient: build_initial_monmap 2024-03-25T19:46:53.970 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.969+0000 7f58e159d640 10 monclient: monmap: 2024-03-25T19:46:53.970 INFO:teuthology.misc.health.smithi044.stderr:epoch 0 2024-03-25T19:46:53.970 INFO:teuthology.misc.health.smithi044.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:53.970 INFO:teuthology.misc.health.smithi044.stderr:last_changed 0.000000 2024-03-25T19:46:53.970 INFO:teuthology.misc.health.smithi044.stderr:created 0.000000 2024-03-25T19:46:53.971 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:53.971 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 1 2024-03-25T19:46:53.971 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.noname-a 2024-03-25T19:46:53.971 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.noname-b 2024-03-25T19:46:53.971 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.noname-c 2024-03-25T19:46:53.971 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:53.971 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.970+0000 7f58e159d640 1 librados: starting msgr at 2024-03-25T19:46:53.971 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.970+0000 7f58e159d640 1 librados: starting objecter 2024-03-25T19:46:53.971 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.970+0000 7f58e159d640 1 Processor -- start 2024-03-25T19:46:53.971 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.970+0000 7f58e159d640 1 -- start start 2024-03-25T19:46:53.971 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.970+0000 7f58e159d640 1 librados: setting wanted keys 2024-03-25T19:46:53.971 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.970+0000 7f58e159d640 1 librados: calling monclient init 2024-03-25T19:46:53.971 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.970+0000 7f58e159d640 10 monclient: init 2024-03-25T19:46:53.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.970+0000 7f58e159d640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:53.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.970+0000 7f58e159d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:53.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.971+0000 7f58e159d640 10 monclient(hunting): picked mon.noname-a con 0x7f58dc0a98f0 addr v1:172.21.15.44:6789/0 2024-03-25T19:46:53.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.971+0000 7f58e159d640 10 monclient(hunting): picked mon.noname-b con 0x7f58dc13f710 addr v1:172.21.15.111:6789/0 2024-03-25T19:46:53.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.971+0000 7f58e159d640 10 monclient(hunting): picked mon.noname-c con 0x7f58dc1430c0 addr v1:172.21.15.44:6790/0 2024-03-25T19:46:53.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.971+0000 7f58e159d640 1 -- --> v1:172.21.15.44:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58dc1384b0 con 0x7f58dc0a98f0 2024-03-25T19:46:53.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.971+0000 7f58e159d640 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58dc138110 con 0x7f58dc13f710 2024-03-25T19:46:53.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.971+0000 7f58e159d640 1 -- --> v1:172.21.15.44:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58dc1389d0 con 0x7f58dc1430c0 2024-03-25T19:46:53.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.971+0000 7f58e159d640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:53.972 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.971+0000 7f58e159d640 10 monclient(hunting): authenticate will time out at 6078.141925s 2024-03-25T19:46:53.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.976+0000 7f58da7fc640 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f58dc13f710 0x7f58dc1d7500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.44:49884/0 (socket says 172.21.15.44:49884) 2024-03-25T19:46:53.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.976+0000 7f58da7fc640 1 -- 172.21.15.44:0/1535341073 learned_addr learned my addr 172.21.15.44:0/1535341073 (peer_addr_for_me v1:172.21.15.44:0/0) 2024-03-25T19:46:53.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.976+0000 7f58c37fe640 10 client.?.objecter ms_handle_connect 0x7f58dc13f710 2024-03-25T19:46:53.976 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.976+0000 7f58c37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:53.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.979+0000 7f58c37fe640 10 client.?.objecter ms_handle_connect 0x7f58dc1430c0 2024-03-25T19:46:53.979 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.979+0000 7f58c37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:53.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.989+0000 7f58c37fe640 1 -- 172.21.15.44:0/1535341073 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291445167 0 0) 0x7f58dc138110 con 0x7f58dc13f710 2024-03-25T19:46:53.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.989+0000 7f58c37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:53.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.989+0000 7f58c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:53.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.989+0000 7f58c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:53.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.989+0000 7f58c37fe640 10 monclient(hunting): my global_id is 4741 2024-03-25T19:46:53.989 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:53.989+0000 7f58c37fe640 1 -- 172.21.15.44:0/1535341073 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58c8004eb0 con 0x7f58dc13f710 2024-03-25T19:46:54.003 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.003+0000 7f58c37fe640 1 -- 172.21.15.44:0/1535341073 <== mon.2 v1:172.21.15.44:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3735006961 0 0) 0x7f58dc1389d0 con 0x7f58dc1430c0 2024-03-25T19:46:54.004 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.003+0000 7f58c37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:54.004 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.003+0000 7f58c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:54.004 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.003+0000 7f58c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:54.004 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.003+0000 7f58c37fe640 10 monclient(hunting): my global_id is 4733 2024-03-25T19:46:54.004 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.003+0000 7f58c37fe640 1 -- 172.21.15.44:0/1535341073 --> v1:172.21.15.44:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58dc138110 con 0x7f58dc1430c0 2024-03-25T19:46:54.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.006+0000 7f58c37fe640 10 client.?.objecter ms_handle_connect 0x7f58dc0a98f0 2024-03-25T19:46:54.006 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.006+0000 7f58c37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:54.014 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.014+0000 7f58c37fe640 1 -- 172.21.15.44:0/1535341073 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1122559007 0 0) 0x7f58c8004eb0 con 0x7f58dc13f710 2024-03-25T19:46:54.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.014+0000 7f58c37fe640 1 -- 172.21.15.44:0/1535341073 --> v1:172.21.15.111:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58dc1389d0 con 0x7f58dc13f710 2024-03-25T19:46:54.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.014+0000 7f58c37fe640 1 -- 172.21.15.44:0/1535341073 <== mon.1 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f58d00047e0 con 0x7f58dc13f710 2024-03-25T19:46:54.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.014+0000 7f58c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:54.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.014+0000 7f58c37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:54.015 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.014+0000 7f58c37fe640 10 monclient(hunting): dump: 2024-03-25T19:46:54.015 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:54.015 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:54.015 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:54.015 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:54.015 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:54.015 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:54.015 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:54.015 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:54.015 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:54.016 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:54.016 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.014+0000 7f58c37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:54.022 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.022+0000 7f58c37fe640 1 -- 172.21.15.44:0/1535341073 <== mon.0 v1:172.21.15.44:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180060027 0 0) 0x7f58dc1384b0 con 0x7f58dc0a98f0 2024-03-25T19:46:54.022 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.022+0000 7f58c37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:54.022 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.022+0000 7f58c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:54.022 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.022+0000 7f58c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:54.022 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.022+0000 7f58c37fe640 10 monclient(hunting): my global_id is 4740 2024-03-25T19:46:54.022 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.022+0000 7f58c37fe640 1 -- 172.21.15.44:0/1535341073 --> v1:172.21.15.44:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58c8004eb0 con 0x7f58dc0a98f0 2024-03-25T19:46:54.024 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.023+0000 7f58c37fe640 1 -- 172.21.15.44:0/1535341073 <== mon.1 v1:172.21.15.111:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4200385159 0 0) 0x7f58dc1389d0 con 0x7f58dc13f710 2024-03-25T19:46:54.024 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.023+0000 7f58c37fe640 1 -- 172.21.15.44:0/1535341073 >> v1:172.21.15.44:6790/0 conn(0x7f58dc1430c0 legacy=0x7f58dc0b0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:54.024 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.023+0000 7f58c37fe640 1 -- 172.21.15.44:0/1535341073 >> v1:172.21.15.44:6789/0 conn(0x7f58dc0a98f0 legacy=0x7f58dc137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:54.024 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.023+0000 7f58c37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:46:54.024 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.023+0000 7f58c37fe640 1 monclient: found mon.b 2024-03-25T19:46:54.024 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.023+0000 7f58c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:54.024 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.023+0000 7f58c37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:46:54.024 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.023+0000 7f58c37fe640 1 -- 172.21.15.44:0/1535341073 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58dc138b60 con 0x7f58dc13f710 2024-03-25T19:46:54.024 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.023+0000 7f58c37fe640 10 monclient: _finish_auth 0 2024-03-25T19:46:54.024 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.023+0000 7f58c37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:46:54.024 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.023+0000 7f58c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:54.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.023+0000 7f58e159d640 5 monclient: authenticate success, global_id 4741 2024-03-25T19:46:54.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.023+0000 7f58e159d640 10 monclient: _renew_subs 2024-03-25T19:46:54.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.023+0000 7f58e159d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:46:54.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.023+0000 7f58e159d640 1 -- 172.21.15.44:0/1535341073 --> v1:172.21.15.111:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58dc1d7c10 con 0x7f58dc13f710 2024-03-25T19:46:54.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.023+0000 7f58e159d640 10 client.4741.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:54.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.023+0000 7f58e159d640 10 monclient: _renew_subs 2024-03-25T19:46:54.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.023+0000 7f58e159d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:46:54.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.023+0000 7f58e159d640 1 -- 172.21.15.44:0/1535341073 --> v1:172.21.15.111:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58dc0b0af0 con 0x7f58dc13f710 2024-03-25T19:46:54.025 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.023+0000 7f58e159d640 1 librados: init done 2024-03-25T19:46:54.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.024+0000 7f58e159d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:54.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.024+0000 7f58e159d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:54.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.024+0000 7f58e159d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:46:54.026 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.024+0000 7f58e159d640 1 -- 172.21.15.44:0/1535341073 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58dc138640 con 0x7f58dc13f710 2024-03-25T19:46:54.243 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.243+0000 7f58c37fe640 1 -- 172.21.15.44:0/1535341073 <== mon.1 v1:172.21.15.111:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58d0001160 con 0x7f58dc13f710 2024-03-25T19:46:54.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.243+0000 7f58c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:54.244 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.243+0000 7f58d97fa640 4 set_mon_vals no callback set 2024-03-25T19:46:54.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.285+0000 7f58c37fe640 1 -- 172.21.15.44:0/1535341073 <== mon.1 v1:172.21.15.111:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1598004030 0 0) 0x7f58d0004d90 con 0x7f58dc13f710 2024-03-25T19:46:54.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.285+0000 7f58c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:54.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.285+0000 7f58c37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:54.285 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.285+0000 7f58c37fe640 10 monclient: dump: 2024-03-25T19:46:54.285 INFO:teuthology.misc.health.smithi044.stderr:epoch 1 2024-03-25T19:46:54.285 INFO:teuthology.misc.health.smithi044.stderr:fsid f7c5aa74-cc2e-41df-a7e9-22cad1e45c88 2024-03-25T19:46:54.285 INFO:teuthology.misc.health.smithi044.stderr:last_changed 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:54.285 INFO:teuthology.misc.health.smithi044.stderr:created 2024-03-25T19:34:36.331446+0000 2024-03-25T19:46:54.285 INFO:teuthology.misc.health.smithi044.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:54.285 INFO:teuthology.misc.health.smithi044.stderr:election_strategy: 3 2024-03-25T19:46:54.286 INFO:teuthology.misc.health.smithi044.stderr:0: v1:172.21.15.44:6789/0 mon.a 2024-03-25T19:46:54.286 INFO:teuthology.misc.health.smithi044.stderr:1: v1:172.21.15.111:6789/0 mon.b 2024-03-25T19:46:54.286 INFO:teuthology.misc.health.smithi044.stderr:2: v1:172.21.15.44:6790/0 mon.c 2024-03-25T19:46:54.286 INFO:teuthology.misc.health.smithi044.stderr: 2024-03-25T19:46:54.287 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.287+0000 7f58c37fe640 1 -- 172.21.15.44:0/1535341073 <== mon.1 v1:172.21.15.111:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 818765503 0 0) 0x7f58d001ade0 con 0x7f58dc13f710 2024-03-25T19:46:54.287 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.288+0000 7f58daffd640 10 monclient: get_auth_request con 0x7f58c806de00 auth_method 0 2024-03-25T19:46:54.288 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.288+0000 7f58c37fe640 10 client.4741.objecter ms_handle_connect 0x7f58c806de00 2024-03-25T19:46:54.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.289+0000 7f58c37fe640 1 -- 172.21.15.44:0/1535341073 <== mon.1 v1:172.21.15.111:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1275227418 0 0) 0x7f58d0054a10 con 0x7f58dc13f710 2024-03-25T19:46:54.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.290+0000 7f58c37fe640 10 client.4741.objecter ms_dispatch 0x7f58dc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:46:54.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.290+0000 7f58c37fe640 3 client.4741.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:46:54.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.290+0000 7f58c37fe640 3 client.4741.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:46:54.290 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.290+0000 7f58c37fe640 20 client.4741.objecter dump_active .. 0 homeless 2024-03-25T19:46:54.292 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.292+0000 7f58c37fe640 1 -- 172.21.15.44:0/1535341073 <== mon.1 v1:172.21.15.111:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f58d0050350 con 0x7f58dc13f710 2024-03-25T19:46:54.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.292+0000 7f58c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:54.293 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.292+0000 7f58c37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:54.549 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.549+0000 7f58e159d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:54.549 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.549+0000 7f58e159d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:54.549 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.549+0000 7f58e159d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.111:6789/0 2024-03-25T19:46:54.550 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.549+0000 7f58e159d640 1 -- 172.21.15.44:0/1535341073 --> v1:172.21.15.111:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58dc1382a0 con 0x7f58dc13f710 2024-03-25T19:46:54.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.581+0000 7f58c37fe640 1 -- 172.21.15.44:0/1535341073 <== mon.1 v1:172.21.15.111:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2782837909) 0x7f58d0052450 con 0x7f58dc13f710 2024-03-25T19:46:54.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.581+0000 7f58c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:54.582 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.581+0000 7f58c37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:54.582 INFO:teuthology.misc.health.smithi044.stdout: 2024-03-25T19:46:54.582 INFO:teuthology.misc.health.smithi044.stdout:{"epoch":11,"fsid":"f7c5aa74-cc2e-41df-a7e9-22cad1e45c88","created":"2024-03-25T19:35:26.477780+0000","modified":"2024-03-25T19:45:30.548832+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:45:30.548832+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6c91f3d4-f50d-4bf8-92a7-d50969121e79","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"24b49d5b-ed7c-4388-a11d-0c292ef38dbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"b6f34510-1d0d-46d1-8e66-17fd9dd62e77","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c938187c-9648-474d-896c-4e91867dfc72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3616d04d-8397-4d67-afb7-b5f914f4a7bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09f68-7386-4523-bf9f-f14877e2e746","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0c107cc3-7f87-4aa3-b3ea-619f471ae849","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f504270b-9a52-449e-8e05-e65bde3085cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:54.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.584+0000 7f58e159d640 10 librados: watch_flush enter 2024-03-25T19:46:54.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.584+0000 7f58e159d640 10 librados: watch_flush exit 2024-03-25T19:46:54.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.584+0000 7f58e159d640 20 client.4741.objecter shutdown clearing up homeless session... 2024-03-25T19:46:54.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.584+0000 7f58e159d640 10 client.4741.objecter successfully canceled tick 2024-03-25T19:46:54.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.584+0000 7f58e159d640 1 -- 172.21.15.44:0/1535341073 >> v1:172.21.15.111:6800/3883215793 conn(0x7f58c806de00 legacy=0x7f58c80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:54.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.584+0000 7f58e159d640 10 monclient: shutdown 2024-03-25T19:46:54.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.584+0000 7f58e159d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:54.585 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.584+0000 7f58e159d640 1 -- 172.21.15.44:0/1535341073 >> v1:172.21.15.111:6789/0 conn(0x7f58dc13f710 legacy=0x7f58dc1d7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:54.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.584+0000 7f58e159d640 1 -- 172.21.15.44:0/1535341073 shutdown_connections 2024-03-25T19:46:54.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.584+0000 7f58e159d640 1 -- 172.21.15.44:0/1535341073 >> 172.21.15.44:0/1535341073 conn(0x7f58dc0b7ea0 msgr2=0x7f58dc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:54.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.584+0000 7f58e159d640 1 -- 172.21.15.44:0/1535341073 shutdown_connections 2024-03-25T19:46:54.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.584+0000 7f58e159d640 1 -- 172.21.15.44:0/1535341073 wait complete. 2024-03-25T19:46:54.586 INFO:teuthology.misc.health.smithi044.stderr:2024-03-25T19:46:54.584+0000 7f58e159d640 1 librados: shutdown 2024-03-25T19:46:54.596 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:54.596 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T19:46:54.598 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T19:46:54.598 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T19:46:54.598 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:46:56.470 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:11:24.562 35566== 2024-03-25T19:46:56.470 INFO:tasks.ceph.osd.0.smithi044.stderr:==00:00:11:24.562 35566== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:46:56.506 INFO:tasks.ceph.osd.0.smithi044.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:47:00.700 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:47:00.700 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi044 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T19:47:00.701 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T19:47:00.701 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T19:47:00.701 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:47:02.604 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:11:30.694 35567== 2024-03-25T19:47:02.604 INFO:tasks.ceph.osd.1.smithi044.stderr:==00:00:11:30.694 35567== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:47:02.641 INFO:tasks.ceph.osd.1.smithi044.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:47:06.199 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:11:57.100 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:06.208 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:11:57.110 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:06.230 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:11:57.132 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:06.255 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:11:57.134 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:06.263 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:11:57.142 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:06.263 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:11:57.165 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:06.266 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:11:57.168 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:06.291 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:11:57.171 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:06.302 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:11:57.204 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:06.802 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:47:06.803 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi044 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T19:47:06.803 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T19:47:06.803 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T19:47:06.803 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:47:08.592 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:11:36.693 35565== 2024-03-25T19:47:08.592 INFO:tasks.ceph.osd.2.smithi044.stderr:==00:00:11:36.693 35565== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:47:08.631 INFO:tasks.ceph.osd.2.smithi044.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:47:12.905 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:47:12.905 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi044 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T19:47:12.906 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T19:47:12.906 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T19:47:12.906 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:47:14.677 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:11:42.763 35568== 2024-03-25T19:47:14.678 INFO:tasks.ceph.osd.3.smithi044.stderr:==00:00:11:42.763 35568== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:47:14.712 INFO:tasks.ceph.osd.3.smithi044.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:47:19.008 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:47:19.008 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi044 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T19:47:19.009 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T19:47:19.009 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T19:47:19.009 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:47:20.882 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:11:48.993 35554== 2024-03-25T19:47:20.882 INFO:tasks.ceph.osd.4.smithi111.stderr:==00:00:11:48.993 35554== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:47:20.923 INFO:tasks.ceph.osd.4.smithi111.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:47:25.110 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:47:25.111 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi111 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T19:47:25.111 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T19:47:25.112 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T19:47:25.112 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:47:26.232 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:12:17.134 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:26.238 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:12:17.140 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:26.262 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:12:17.164 35252== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:26.290 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:12:17.169 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:26.292 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:12:17.171 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:26.298 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:12:17.200 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:26.302 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:12:17.204 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:26.308 INFO:tasks.ceph.mon.b.smithi111.stderr:==00:00:12:17.188 35010== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:26.325 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:12:17.226 35254== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:26.901 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:11:55.010 35555== 2024-03-25T19:47:26.901 INFO:tasks.ceph.osd.5.smithi111.stderr:==00:00:11:55.010 35555== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:47:26.942 INFO:tasks.ceph.osd.5.smithi111.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:47:31.214 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:47:31.214 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi111 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T19:47:31.214 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T19:47:31.215 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T19:47:31.215 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:47:33.045 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:12:01.152 35556== 2024-03-25T19:47:33.045 INFO:tasks.ceph.osd.6.smithi111.stderr:==00:00:12:01.152 35556== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:47:33.091 INFO:tasks.ceph.osd.6.smithi111.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:47:37.317 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:47:37.317 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi111 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T19:47:37.317 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T19:47:37.318 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T19:47:37.318 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:47:38.743 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:12:06.849 35557== 2024-03-25T19:47:38.744 INFO:tasks.ceph.osd.7.smithi111.stderr:==00:00:12:06.849 35557== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:47:38.795 INFO:tasks.ceph.osd.7.smithi111.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:47:43.419 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:47:43.420 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi111 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T19:47:43.420 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T19:47:43.420 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T19:47:43.420 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T19:47:43.421 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:47:43.445 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T19:47:43.445 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T19:47:43.446 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:47:43.447 INFO:tasks.ceph.mgr.x.smithi111.stderr:2024-03-25T19:47:43.447+0000 7f881bebb640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 35008) UID: 0 2024-03-25T19:47:43.472 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T19:47:43.472 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T19:47:43.472 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T19:47:43.472 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:47:43.503 INFO:tasks.ceph.mon.a.smithi044.stderr:2024-03-25T19:47:43.490+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35220) UID: 0 2024-03-25T19:47:43.503 INFO:tasks.ceph.mon.a.smithi044.stderr:2024-03-25T19:47:43.492+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T19:47:44.054 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:12:34.955 35252== 2024-03-25T19:47:44.054 INFO:tasks.ceph.mon.a.smithi044.stderr:==00:00:12:34.955 35252== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:47:44.064 INFO:tasks.ceph.mon.a.smithi044.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:47:49.574 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:47:49.574 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi044 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T19:47:49.575 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T19:47:49.575 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T19:47:49.575 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:47:49.593 INFO:tasks.ceph.mon.c.smithi044.stderr:2024-03-25T19:47:49.579+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35222) UID: 0 2024-03-25T19:47:49.593 INFO:tasks.ceph.mon.c.smithi044.stderr:2024-03-25T19:47:49.581+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T19:52:48.645 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T19:52:48.646 DEBUG:teuthology.orchestra.run.smithi044:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T19:52:48.688 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi044.front.sepia.ceph.com 2024-03-25T19:52:48.688 DEBUG:teuthology.orchestra.run.smithi044:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T19:52:48.782 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi044.front.sepia.ceph.com 2024-03-25T19:52:48.782 DEBUG:teuthology.orchestra.run.smithi044:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T19:52:48.830 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi044.front.sepia.ceph.com 2024-03-25T19:52:48.830 DEBUG:teuthology.orchestra.run.smithi044:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T19:52:48.882 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi044.front.sepia.ceph.com 2024-03-25T19:52:48.882 DEBUG:teuthology.orchestra.run.smithi044:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T19:52:48.931 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi111.front.sepia.ceph.com 2024-03-25T19:52:48.931 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T19:52:49.071 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi111.front.sepia.ceph.com 2024-03-25T19:52:49.072 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T19:52:49.118 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi111.front.sepia.ceph.com 2024-03-25T19:52:49.118 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T19:52:49.167 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi111.front.sepia.ceph.com 2024-03-25T19:52:49.167 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T19:52:49.218 INFO:tasks.ceph:Archiving mon data... 2024-03-25T19:52:49.219 DEBUG:teuthology.misc:Transferring archived files from smithi044:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621321/data/mon.a.tgz 2024-03-25T19:52:49.219 DEBUG:teuthology.orchestra.run.smithi044:> mktemp 2024-03-25T19:52:49.233 INFO:teuthology.orchestra.run.smithi044.stdout:/tmp/tmp.Dvvw9bkk0p 2024-03-25T19:52:49.233 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Dvvw9bkk0p 2024-03-25T19:52:49.330 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0666 /tmp/tmp.Dvvw9bkk0p 2024-03-25T19:52:49.412 DEBUG:teuthology.orchestra.remote:smithi044:/tmp/tmp.Dvvw9bkk0p is 133KB 2024-03-25T19:52:49.463 DEBUG:teuthology.orchestra.run.smithi044:> rm -fr /tmp/tmp.Dvvw9bkk0p 2024-03-25T19:52:49.476 DEBUG:teuthology.misc:Transferring archived files from smithi044:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621321/data/mon.c.tgz 2024-03-25T19:52:49.476 DEBUG:teuthology.orchestra.run.smithi044:> mktemp 2024-03-25T19:52:49.529 INFO:teuthology.orchestra.run.smithi044.stdout:/tmp/tmp.BGnpOcm0oI 2024-03-25T19:52:49.529 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.BGnpOcm0oI 2024-03-25T19:52:49.617 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0666 /tmp/tmp.BGnpOcm0oI 2024-03-25T19:52:49.699 DEBUG:teuthology.orchestra.remote:smithi044:/tmp/tmp.BGnpOcm0oI is 133KB 2024-03-25T19:52:49.749 DEBUG:teuthology.orchestra.run.smithi044:> rm -fr /tmp/tmp.BGnpOcm0oI 2024-03-25T19:52:49.763 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621321/data/mon.b.tgz 2024-03-25T19:52:49.763 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2024-03-25T19:52:49.780 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.scnnZ55UVS 2024-03-25T19:52:49.780 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.scnnZ55UVS 2024-03-25T19:52:49.890 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.scnnZ55UVS 2024-03-25T19:52:49.977 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.scnnZ55UVS is 134KB 2024-03-25T19:52:50.028 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.scnnZ55UVS 2024-03-25T19:52:50.043 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T19:52:50.043 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:52:50.045 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:52:50.228 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 52, column 1 (, line 52) 2024-03-25T19:52:50.420 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:52:50.493 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 52, column 1 (, line 52) 2024-03-25T19:52:50.687 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T19:52:50.690 DEBUG:teuthology.misc:Transferring archived files from smithi044:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621321/remote/smithi044/crash 2024-03-25T19:52:50.690 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:52:50.716 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621321/remote/smithi111/crash 2024-03-25T19:52:50.717 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:52:50.743 INFO:tasks.ceph:Compressing logs... 2024-03-25T19:52:50.743 DEBUG:teuthology.orchestra.run.smithi044:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:52:50.761 DEBUG:teuthology.orchestra.run.smithi111:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:52:50.827 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40740.log 2024-03-25T19:52:50.827 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35591.log 2024-03-25T19:52:50.827 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40177.log 2024-03-25T19:52:50.827 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.35591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39592.log 2024-03-25T19:52:50.828 INFO:teuthology.orchestra.run.smithi044.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40740.log.gz 2024-03-25T19:52:50.828 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41520.log 2024-03-25T19:52:50.828 INFO:teuthology.orchestra.run.smithi044.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.35591.log.gz 2024-03-25T19:52:50.828 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T19:52:50.828 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39592.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40177.log.gz 2024-03-25T19:52:50.828 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39904.log 2024-03-25T19:52:50.828 INFO:teuthology.orchestra.run.smithi044.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39592.log.gz 2024-03-25T19:52:50.828 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41013.log 2024-03-25T19:52:50.828 INFO:teuthology.orchestra.run.smithi044.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41520.log.gz 2024-03-25T19:52:50.828 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42066.log 2024-03-25T19:52:50.828 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39904.log: /var/log/ceph/ceph-client.admin.41013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39082.log 2024-03-25T19:52:50.828 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.42066.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39904.log.gz 2024-03-25T19:52:50.828 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40466.log 2024-03-25T19:52:50.829 INFO:teuthology.orchestra.run.smithi044.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41013.log.gz 2024-03-25T19:52:50.829 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40216.log 2024-03-25T19:52:50.829 INFO:teuthology.orchestra.run.smithi044.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42066.log.gz 2024-03-25T19:52:50.829 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41793.log 2024-03-25T19:52:50.829 INFO:teuthology.orchestra.run.smithi044.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39082.log.gz 2024-03-25T19:52:50.829 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40216.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T19:52:50.829 INFO:teuthology.orchestra.run.smithi044.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40466.log.gz 2024-03-25T19:52:50.829 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T19:52:50.829 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41793.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40216.log.gz 2024-03-25T19:52:50.829 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T19:52:50.829 INFO:teuthology.orchestra.run.smithi044.stderr: 55.9% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T19:52:50.829 INFO:teuthology.orchestra.run.smithi044.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41793.log.gz 2024-03-25T19:52:50.829 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T19:52:50.829 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/valgrind/osd.0.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T19:52:50.830 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T19:52:50.830 INFO:teuthology.orchestra.run.smithi044.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T19:52:50.830 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T19:52:50.830 INFO:teuthology.orchestra.run.smithi044.stderr: 72.0%gzip -- replaced with /var/log/ceph/valgrind/osd.1.log.gz -5 2024-03-25T19:52:50.830 INFO:teuthology.orchestra.run.smithi044.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39358.log 2024-03-25T19:52:50.830 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/valgrind/mon.a.log: /var/log/ceph/valgrind/osd.3.log: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T19:52:50.830 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41208.log 2024-03-25T19:52:50.830 INFO:teuthology.orchestra.run.smithi044.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T19:52:50.830 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41052.log 2024-03-25T19:52:50.830 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39358.log: /var/log/ceph/ceph-client.admin.41208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36587.log 2024-03-25T19:52:50.830 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41988.log 2024-03-25T19:52:50.830 INFO:teuthology.orchestra.run.smithi044.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39358.log.gz 2024-03-25T19:52:50.830 INFO:teuthology.orchestra.run.smithi044.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41208.log.gz 2024-03-25T19:52:50.831 INFO:teuthology.orchestra.run.smithi044.stderr:gzip/var/log/ceph/ceph-client.admin.36587.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41676.log 2024-03-25T19:52:50.831 INFO:teuthology.orchestra.run.smithi044.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41052.log.gz 2024-03-25T19:52:50.831 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41910.log 2024-03-25T19:52:50.831 INFO:teuthology.orchestra.run.smithi044.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36587.log.gz 2024-03-25T19:52:50.831 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41676.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41988.log.gz 2024-03-25T19:52:50.831 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39670.log 2024-03-25T19:52:50.831 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40935.log 2024-03-25T19:52:50.831 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41910.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41676.log.gz 2024-03-25T19:52:50.831 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41247.log 2024-03-25T19:52:50.831 INFO:teuthology.orchestra.run.smithi044.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41910.log.gz 2024-03-25T19:52:50.831 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39943.log 2024-03-25T19:52:50.831 INFO:teuthology.orchestra.run.smithi044.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39670.log.gz 2024-03-25T19:52:50.831 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41247.log: gzip -5 --verbose 87.3% -- -- replaced with /var/log/ceph/ceph-client.admin.40935.log.gz /var/log/ceph/ceph-client.admin.41169.log 2024-03-25T19:52:50.831 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T19:52:50.831 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39943.log: gzip -5 --verbose 86.3% -- -- replaced with /var/log/ceph/ceph-client.admin.41247.log.gz 2024-03-25T19:52:50.831 INFO:teuthology.orchestra.run.smithi044.stderr: /var/log/ceph/ceph-client.admin.41325.log 2024-03-25T19:52:50.832 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37612.log 2024-03-25T19:52:50.832 INFO:teuthology.orchestra.run.smithi044.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39943.log.gz 2024-03-25T19:52:50.832 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39709.log 2024-03-25T19:52:50.832 INFO:teuthology.orchestra.run.smithi044.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41169.log.gz 2024-03-25T19:52:50.832 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.37612.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:52:50.832 INFO:teuthology.orchestra.run.smithi044.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41325.log.gz 2024-03-25T19:52:50.832 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40505.log 2024-03-25T19:52:50.832 INFO:teuthology.orchestra.run.smithi044.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.37612.log.gz 2024-03-25T19:52:50.832 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39121.log 2024-03-25T19:52:50.832 INFO:teuthology.orchestra.run.smithi044.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39709.log.gz 2024-03-25T19:52:50.832 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41754.log 2024-03-25T19:52:50.832 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39121.log: gzip -5 93.4% --verbose -- -- replaced with /var/log/ceph/ceph.audit.log.gz /var/log/ceph/ceph-client.admin.40138.log 2024-03-25T19:52:50.832 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T19:52:50.832 INFO:teuthology.orchestra.run.smithi044.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40505.log.gz 2024-03-25T19:52:50.832 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.41754.log: /var/log/ceph/ceph-client.admin.39865.log 2024-03-25T19:52:50.832 INFO:teuthology.orchestra.run.smithi044.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39121.log.gz 2024-03-25T19:52:50.833 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40021.log 2024-03-25T19:52:50.833 INFO:teuthology.orchestra.run.smithi044.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41754.log.gz 2024-03-25T19:52:50.833 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39865.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40138.log.gzgzip 2024-03-25T19:52:50.833 INFO:teuthology.orchestra.run.smithi044.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40099.log 2024-03-25T19:52:50.833 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41481.log 2024-03-25T19:52:50.833 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40021.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39865.log.gz 2024-03-25T19:52:50.833 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40099.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T19:52:50.833 INFO:teuthology.orchestra.run.smithi044.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40021.log.gz 2024-03-25T19:52:50.833 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41559.log 2024-03-25T19:52:50.833 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41481.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40099.log.gz 2024-03-25T19:52:50.833 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39280.log 2024-03-25T19:52:50.833 INFO:teuthology.orchestra.run.smithi044.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41481.log.gz 2024-03-25T19:52:50.833 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42183.log 2024-03-25T19:52:50.833 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42105.log 2024-03-25T19:52:50.833 INFO:teuthology.orchestra.run.smithi044.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41559.log.gz 2024-03-25T19:52:50.834 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.42183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41949.log 2024-03-25T19:52:50.834 INFO:teuthology.orchestra.run.smithi044.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39280.log.gz 2024-03-25T19:52:50.834 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.42105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40622.log 2024-03-25T19:52:50.834 INFO:teuthology.orchestra.run.smithi044.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42183.log.gz 2024-03-25T19:52:50.834 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40060.log 2024-03-25T19:52:50.834 INFO:teuthology.orchestra.run.smithi044.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42105.log.gz 2024-03-25T19:52:50.834 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40544.log 2024-03-25T19:52:50.834 INFO:teuthology.orchestra.run.smithi044.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41949.log.gz 2024-03-25T19:52:50.834 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40060.log: gzip -5 --verbose 86.6% -- /var/log/ceph/ceph-client.admin.41832.log 2024-03-25T19:52:50.834 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40622.log.gz 2024-03-25T19:52:50.834 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40896.log 2024-03-25T19:52:50.834 INFO:teuthology.orchestra.run.smithi044.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40060.log.gz 2024-03-25T19:52:50.834 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35732.log 2024-03-25T19:52:50.834 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41832.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40544.log.gz 2024-03-25T19:52:50.835 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39826.log 2024-03-25T19:52:50.835 INFO:teuthology.orchestra.run.smithi044.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41832.log.gz 2024-03-25T19:52:50.835 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.35732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42222.log 2024-03-25T19:52:50.835 INFO:teuthology.orchestra.run.smithi044.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40896.log.gz 2024-03-25T19:52:50.835 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39826.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T19:52:50.835 INFO:teuthology.orchestra.run.smithi044.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35732.log.gz 2024-03-25T19:52:50.835 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.42222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40857.log 2024-03-25T19:52:50.835 INFO:teuthology.orchestra.run.smithi044.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39826.log.gz 2024-03-25T19:52:50.835 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-osd.3.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42222.log.gz 2024-03-25T19:52:50.835 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39787.log 2024-03-25T19:52:50.835 INFO:teuthology.orchestra.run.smithi044.stderr:gzip/var/log/ceph/ceph-client.admin.40857.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39475.log 2024-03-25T19:52:50.835 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39787.log: gzip -5 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40857.log.gz 2024-03-25T19:52:50.835 INFO:teuthology.orchestra.run.smithi044.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39397.log 2024-03-25T19:52:50.835 INFO:teuthology.orchestra.run.smithi044.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39787.log.gz 2024-03-25T19:52:50.835 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42378.log 2024-03-25T19:52:50.836 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39397.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39475.log.gz 2024-03-25T19:52:50.836 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39553.log 2024-03-25T19:52:50.836 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.42378.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39397.log.gz 2024-03-25T19:52:50.836 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40818.log 2024-03-25T19:52:50.836 INFO:teuthology.orchestra.run.smithi044.stderr:gzip/var/log/ceph/ceph-client.admin.39553.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41403.log 2024-03-25T19:52:50.836 INFO:teuthology.orchestra.run.smithi044.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.42378.log.gz 2024-03-25T19:52:50.836 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35564.log 2024-03-25T19:52:50.836 INFO:teuthology.orchestra.run.smithi044.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39553.log.gz/var/log/ceph/ceph-client.admin.41403.log: 2024-03-25T19:52:50.836 INFO:teuthology.orchestra.run.smithi044.stderr: 86.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42144.log 2024-03-25T19:52:50.836 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40818.log.gz 2024-03-25T19:52:50.836 INFO:teuthology.orchestra.run.smithi044.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41403.log.gz 2024-03-25T19:52:50.836 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.35564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40779.log 2024-03-25T19:52:50.836 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.42144.log: /var/log/ceph/ceph-client.admin.39631.log 85.8% -- replaced with /var/log/ceph/ceph-client.admin.35564.log.gz 2024-03-25T19:52:50.836 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T19:52:50.837 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40779.log: gzip -5 --verbose 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42144.log.gz 2024-03-25T19:52:50.837 INFO:teuthology.orchestra.run.smithi044.stderr: -- /var/log/ceph/ceph-client.admin.39436.log 2024-03-25T19:52:50.837 INFO:teuthology.orchestra.run.smithi044.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40779.log.gz 2024-03-25T19:52:50.837 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38867.log 2024-03-25T19:52:50.837 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39436.log: 86.2%gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T19:52:50.837 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39631.log.gz 2024-03-25T19:52:50.837 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.38867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41598.log 2024-03-25T19:52:50.837 INFO:teuthology.orchestra.run.smithi044.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39436.log.gz 2024-03-25T19:52:50.837 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose --/var/log/ceph/ceph-mgr.y.log: /var/log/ceph/ceph-client.admin.39319.log 2024-03-25T19:52:50.837 INFO:teuthology.orchestra.run.smithi044.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38867.log.gz 2024-03-25T19:52:50.837 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42027.log 2024-03-25T19:52:50.837 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40974.log 2024-03-25T19:52:50.837 INFO:teuthology.orchestra.run.smithi044.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41598.log.gz 2024-03-25T19:52:50.837 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39319.log: /var/log/ceph/ceph-client.admin.42027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39241.log 2024-03-25T19:52:50.838 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40974.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39319.log.gz 2024-03-25T19:52:50.838 INFO:teuthology.orchestra.run.smithi044.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42027.log.gz 2024-03-25T19:52:50.838 INFO:teuthology.orchestra.run.smithi044.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40974.log.gz 2024-03-25T19:52:50.838 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40583.log 2024-03-25T19:52:50.838 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41286.log 2024-03-25T19:52:50.838 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39241.log: /var/log/ceph/ceph-client.admin.40583.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T19:52:50.838 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41286.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39241.log.gz 2024-03-25T19:52:50.838 INFO:teuthology.orchestra.run.smithi044.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40583.log.gz 2024-03-25T19:52:50.838 INFO:teuthology.orchestra.run.smithi044.stderr: 86.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40255.log 2024-03-25T19:52:50.838 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41286.log.gz 2024-03-25T19:52:50.838 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40347.log 2024-03-25T19:52:50.838 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40427.log 2024-03-25T19:52:50.839 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40347.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40255.log.gz 2024-03-25T19:52:50.839 INFO:teuthology.orchestra.run.smithi044.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40347.log.gz 2024-03-25T19:52:50.839 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40386.log 2024-03-25T19:52:50.839 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:52:50.839 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40386.log: /var/log/ceph/ceph-client.admin.40427.log: 86.3% 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40427.log.gz 2024-03-25T19:52:50.839 INFO:teuthology.orchestra.run.smithi044.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40386.log.gz 2024-03-25T19:52:50.839 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40661.log 2024-03-25T19:52:50.839 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40700.log 2024-03-25T19:52:50.839 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40661.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40661.log.gz 2024-03-25T19:52:50.839 INFO:teuthology.orchestra.run.smithi044.stderr: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:52:50.839 INFO:teuthology.orchestra.run.smithi044.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T19:52:50.839 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29409.log 2024-03-25T19:52:50.839 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39162.log 2024-03-25T19:52:50.840 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph.tmp-client.admin.29409.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40700.log.gz 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29409.log.gz 2024-03-25T19:52:50.840 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T19:52:50.840 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T19:52:50.840 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35262.log 2024-03-25T19:52:50.840 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39162.log: /var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41715.log 2024-03-25T19:52:50.840 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.35262.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39162.log.gz 2024-03-25T19:52:50.840 INFO:teuthology.orchestra.run.smithi044.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.35262.log.gz 2024-03-25T19:52:50.846 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T19:52:50.846 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35455.log 2024-03-25T19:52:50.847 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35284.log 2024-03-25T19:52:50.847 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.35455.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T19:52:50.847 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.35284.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T19:52:50.847 INFO:teuthology.orchestra.run.smithi111.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.35455.log.gz 2024-03-25T19:52:50.847 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T19:52:50.847 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/valgrind/osd.7.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35284.log.gz 2024-03-25T19:52:50.847 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/valgrind/osd.5.log: 72.1% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T19:52:50.847 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T19:52:50.847 INFO:teuthology.orchestra.run.smithi111.stderr: 77.3% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T19:52:50.847 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T19:52:50.847 INFO:teuthology.orchestra.run.smithi111.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T19:52:50.847 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35164.log 2024-03-25T19:52:50.847 INFO:teuthology.orchestra.run.smithi111.stderr: 55.8% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T19:52:50.848 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35244.log 2024-03-25T19:52:50.848 INFO:teuthology.orchestra.run.smithi111.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T19:52:50.848 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.35164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35364.log 2024-03-25T19:52:50.848 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.35244.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:52:50.848 INFO:teuthology.orchestra.run.smithi111.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.35164.log.gz 2024-03-25T19:52:50.848 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.35364.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T19:52:50.848 INFO:teuthology.orchestra.run.smithi111.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35244.log.gz 2024-03-25T19:52:50.848 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35414.log 2024-03-25T19:52:50.848 INFO:teuthology.orchestra.run.smithi111.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35364.log.gz 2024-03-25T19:52:50.848 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T19:52:50.848 INFO:teuthology.orchestra.run.smithi111.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:52:50.848 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.35414.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T19:52:50.848 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35324.log 2024-03-25T19:52:50.848 INFO:teuthology.orchestra.run.smithi111.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35414.log.gz 2024-03-25T19:52:50.849 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35204.log 2024-03-25T19:52:50.849 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.35324.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T19:52:50.849 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.35204.log: gzip -5 --verbose 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35324.log.gz 2024-03-25T19:52:50.849 INFO:teuthology.orchestra.run.smithi111.stderr: -- /var/log/ceph/ceph.log 2024-03-25T19:52:50.849 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-mgr.x.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T19:52:50.849 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35204.log.gz 2024-03-25T19:52:50.849 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:52:50.857 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41091.log 2024-03-25T19:52:50.857 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42300.log 2024-03-25T19:52:50.857 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41091.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41715.log.gz 2024-03-25T19:52:50.857 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39514.log 2024-03-25T19:52:50.857 INFO:teuthology.orchestra.run.smithi044.stderr: 86.9%/var/log/ceph/ceph-client.admin.42300.log: -- replaced with /var/log/ceph/ceph-client.admin.41091.log.gz 2024-03-25T19:52:50.857 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- 86.1% -- replaced with /var/log/ceph/ceph-client.admin.42300.log.gz 2024-03-25T19:52:50.857 INFO:teuthology.orchestra.run.smithi044.stderr: /var/log/ceph/ceph-client.admin.39201.log 2024-03-25T19:52:50.857 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41442.log 2024-03-25T19:52:50.857 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39201.log: /var/log/ceph/ceph-client.admin.39514.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39514.log.gz 2024-03-25T19:52:50.857 INFO:teuthology.orchestra.run.smithi044.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39201.log.gz 2024-03-25T19:52:50.857 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39982.log 2024-03-25T19:52:50.857 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41130.log 2024-03-25T19:52:50.857 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41442.log: /var/log/ceph/ceph-client.admin.39982.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41442.log.gz 2024-03-25T19:52:50.857 INFO:teuthology.orchestra.run.smithi044.stderr: 86.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.39982.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.41637.log 2024-03-25T19:52:50.858 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T19:52:50.858 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41871.log 2024-03-25T19:52:50.858 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41130.log: /var/log/ceph/ceph-client.admin.41637.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41130.log.gz 2024-03-25T19:52:50.858 INFO:teuthology.orchestra.run.smithi044.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41637.log.gz 2024-03-25T19:52:50.858 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42261.log 2024-03-25T19:52:50.858 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40294.log 2024-03-25T19:52:50.858 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41871.log: /var/log/ceph/ceph-client.admin.42261.log: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.41871.log.gz 2024-03-25T19:52:50.858 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41364.log 2024-03-25T19:52:50.858 INFO:teuthology.orchestra.run.smithi044.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42261.log.gz 2024-03-25T19:52:50.858 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.40294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42339.log 2024-03-25T19:52:50.858 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.41364.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40294.log.gz 2024-03-25T19:52:50.858 INFO:teuthology.orchestra.run.smithi044.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41364.log.gz 2024-03-25T19:52:50.858 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T19:52:50.859 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39748.log 2024-03-25T19:52:50.859 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.42339.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.42339.log.gz 2024-03-25T19:52:50.859 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.39748.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39748.log.gz 2024-03-25T19:52:50.872 INFO:teuthology.orchestra.run.smithi111.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T19:52:51.698 INFO:teuthology.orchestra.run.smithi111.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T19:52:51.710 INFO:teuthology.orchestra.run.smithi111.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T19:52:51.760 INFO:teuthology.orchestra.run.smithi111.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T19:52:51.991 INFO:teuthology.orchestra.run.smithi111.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T19:52:52.040 INFO:teuthology.orchestra.run.smithi044.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T19:52:52.045 INFO:teuthology.orchestra.run.smithi044.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T19:52:52.060 INFO:teuthology.orchestra.run.smithi044.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T19:52:52.065 INFO:teuthology.orchestra.run.smithi044.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T19:52:56.007 INFO:teuthology.orchestra.run.smithi111.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping 2024-03-25T19:52:56.007 INFO:teuthology.orchestra.run.smithi111.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T19:52:56.009 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-03-25T19:52:56.009 INFO:teuthology.orchestra.run.smithi111.stderr:real 0m5.205s 2024-03-25T19:52:56.009 INFO:teuthology.orchestra.run.smithi111.stderr:user 0m8.886s 2024-03-25T19:52:56.009 INFO:teuthology.orchestra.run.smithi111.stderr:sys 0m0.262s 2024-03-25T19:52:56.073 INFO:teuthology.orchestra.run.smithi044.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T19:52:56.303 INFO:teuthology.orchestra.run.smithi044.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T19:52:56.305 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-03-25T19:52:56.305 INFO:teuthology.orchestra.run.smithi044.stderr:real 0m5.533s 2024-03-25T19:52:56.305 INFO:teuthology.orchestra.run.smithi044.stderr:user 0m15.404s 2024-03-25T19:52:56.305 INFO:teuthology.orchestra.run.smithi044.stderr:sys 0m0.446s 2024-03-25T19:52:56.306 INFO:tasks.ceph:Archiving logs... 2024-03-25T19:52:56.306 DEBUG:teuthology.misc:Transferring archived files from smithi044:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621321/remote/smithi044/log 2024-03-25T19:52:56.307 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:52:57.554 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:17:48.430 35254== 2024-03-25T19:52:57.554 INFO:tasks.ceph.mon.c.smithi044.stderr:==00:00:17:48.430 35254== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:57.554 INFO:tasks.ceph.mon.c.smithi044.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:58.831 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621321/remote/smithi111/log 2024-03-25T19:52:58.832 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:53:00.312 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:53:00.517 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7d0ba20938414f32af97791f2d05c45c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:53:00.522 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T19:53:00.532 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T19:53:00.540 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T19:53:00.540 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:53:00.553 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:53:00.634 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T19:53:00.643 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T19:53:00.643 DEBUG:teuthology.orchestra.run.smithi044:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:53:00.645 DEBUG:teuthology.orchestra.run.smithi111:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:53:00.659 INFO:teuthology.orchestra.run.smithi044.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:53:00.661 INFO:teuthology.orchestra.run.smithi111.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:53:00.665 INFO:teuthology.orchestra.run.smithi044.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:53:00.665 INFO:teuthology.orchestra.run.smithi044.stdout:=============================================================================== 2024-03-25T19:53:00.665 INFO:teuthology.orchestra.run.smithi044.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 25 -535us[ -535us] +/- 78ms 2024-03-25T19:53:00.665 INFO:teuthology.orchestra.run.smithi044.stdout:^* hv02.front.sepia.ceph.com 3 6 377 28 -222us[ -201us] +/- 65ms 2024-03-25T19:53:00.665 INFO:teuthology.orchestra.run.smithi044.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 +726us[ +726us] +/- 68ms 2024-03-25T19:53:00.666 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:53:00.666 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:53:00.667 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-03-25T19:53:00.667 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 29 -494us[ -494us] +/- 78ms 2024-03-25T19:53:00.667 INFO:teuthology.orchestra.run.smithi111.stdout:^* hv02.front.sepia.ceph.com 3 6 377 30 -203us[ -162us] +/- 65ms 2024-03-25T19:53:00.667 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 +757us[ +757us] +/- 68ms 2024-03-25T19:53:00.667 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:53:00.667 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T19:53:00.676 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T19:53:00.676 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T19:53:00.684 DEBUG:teuthology.orchestra.run.smithi044:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:53:00.778 DEBUG:teuthology.orchestra.run.smithi111:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:53:00.850 DEBUG:teuthology.orchestra.run.smithi044:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:53:00.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:53:00.883 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:53:00.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:53:00.914 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T19:53:00.923 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T19:53:00.930 INFO:teuthology.task.internal:Duration was 6070.660284 seconds 2024-03-25T19:53:00.930 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T19:53:00.937 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T19:53:00.938 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:53:00.940 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:53:00.983 INFO:teuthology.orchestra.run.smithi044.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:53:00.999 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:53:01.418 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T19:53:01.418 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi044.front.sepia.ceph.com 2024-03-25T19:53:01.418 DEBUG:teuthology.orchestra.run.smithi044:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:53:01.441 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi111.front.sepia.ceph.com 2024-03-25T19:53:01.442 DEBUG:teuthology.orchestra.run.smithi111:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:53:01.485 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T19:53:01.486 DEBUG:teuthology.orchestra.run.smithi044:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:53:01.488 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:53:01.555 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T19:53:01.555 DEBUG:teuthology.orchestra.run.smithi044:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:53:01.562 DEBUG:teuthology.orchestra.run.smithi111:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:53:01.699 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T19:53:01.709 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T19:53:01.709 DEBUG:teuthology.orchestra.run.smithi044:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:53:01.737 DEBUG:teuthology.orchestra.run.smithi111:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:53:01.763 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T19:53:01.772 DEBUG:teuthology.orchestra.run.smithi044:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:53:01.779 DEBUG:teuthology.orchestra.run.smithi111:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:53:01.802 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern = core 2024-03-25T19:53:01.829 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = core 2024-03-25T19:53:01.850 DEBUG:teuthology.orchestra.run.smithi044:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:53:01.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:53:01.892 DEBUG:teuthology.orchestra.run.smithi111:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:53:01.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:53:01.906 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T19:53:01.915 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T19:53:01.916 DEBUG:teuthology.misc:Transferring archived files from smithi044:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621321/remote/smithi044 2024-03-25T19:53:01.916 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:53:01.969 DEBUG:teuthology.misc:Transferring archived files from smithi111:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621321/remote/smithi111 2024-03-25T19:53:01.969 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:53:02.004 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T19:53:02.005 DEBUG:teuthology.orchestra.run.smithi044:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:53:02.007 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:53:02.054 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T19:53:02.066 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T19:53:02.066 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T19:53:02.075 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T19:53:02.075 DEBUG:teuthology.orchestra.run.smithi044:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:53:02.078 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:53:02.090 INFO:teuthology.orchestra.run.smithi044.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:53 /home/ubuntu/cephtest 2024-03-25T19:53:02.108 INFO:teuthology.orchestra.run.smithi111.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:53 /home/ubuntu/cephtest 2024-03-25T19:53:02.109 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T19:53:02.119 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T19:53:02.138 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T19:53:02.162 DEBUG:teuthology.nuke:shortname: smithi044 2024-03-25T19:53:02.162 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:53:02.182 DEBUG:teuthology.nuke:shortname: smithi111 2024-03-25T19:53:02.182 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:53:02.187 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi044.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621321', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:01:38.261477', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJiZboMgAKUllmKZXbQtFQg4Vo+6OojgRvyCN6U7ovHyVJwE05w3X98t/cBtzDQ24ni5kRFzxbZOCHzqX/RoBeY='} 2024-03-25T19:53:02.204 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621321', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:01:38.260293', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB6T5MuIscewO93LAeMlMw2BXb8rpyN4fq4Q2BO2Ncz/pFnpvRkgRfFEX/tARk77Vj9MbOnGxQm1YbS2Y7/PybQ='} 2024-03-25T19:53:02.230 INFO:teuthology.orchestra.console.smithi044:Power off 2024-03-25T19:53:02.231 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:53:02.256 DEBUG:teuthology.orchestra.console.smithi044:power off output: Chassis Power Control: Down/Off 2024-03-25T19:53:02.257 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:53:02.270 INFO:teuthology.orchestra.console.smithi111:Power off 2024-03-25T19:53:02.271 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:53:02.280 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is on 2024-03-25T19:53:02.295 DEBUG:teuthology.orchestra.console.smithi111:power off output: Chassis Power Control: Down/Off 2024-03-25T19:53:02.296 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:53:02.319 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2024-03-25T19:53:06.281 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:53:06.321 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:53:06.392 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is on 2024-03-25T19:53:06.429 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2024-03-25T19:53:10.397 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:53:10.430 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:53:10.508 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is off 2024-03-25T19:53:10.539 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2024-03-25T19:53:10.609 INFO:teuthology.orchestra.console.smithi044:Power off completed 2024-03-25T19:53:14.542 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:53:14.653 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is off 2024-03-25T19:53:14.753 INFO:teuthology.orchestra.console.smithi111:Power off completed 2024-03-25T19:53:14.860 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 6070.660283565521 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=7d0ba20938414f32af97791f2d05c45c status: fail success: false 2024-03-25T19:53:14.860 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T19:53:14.947 INFO:teuthology.run:FAIL